Automatic robots.txt inclusion

Our performance monitoring and optimization system may automatically add "robots.txt" to one or all of your domain names hosted with us (shared/VPS/Cloud packages), if your domain name lacks robots.txt file when initially deployed. Our system will not alter or replace any existing robots.txt files, regardless if they are properly structured or search engine friendly.

Our default robots.txt is search engine safe and friendly (does not block any specific search engines or user agents), and it helps mitigate load caused by automatic search engine crawlers, especially if you are hosting applications such as WordPress or Joomla. For more information on robots and their impact on your account performance please review:

Using robots.txt in order to optimize your site performance and reduce your website load

Please, note that additional restrictions to how search engine crawlers scan your account maybe enforced if your sites are getting slowed down due to search engine crawlers or your server is running out of resources to handle all incoming requests.