A robots.txt file has a very specific purpose and one company botched it magnificently.
When they realized there was a problem, they called up their SEO agency in a panic. Their website was no longer showing up in Google’s search results.
They frantically asked the agency if they thought they’d been penalized by one of Google’s updates and if they’d ever get their site back on the map.
After some head scratching and a little bit of research, the agency found that they had blocked their entire site from being indexed with their robotx.txt file.
Yes. The ENTIRE site.
Once this little error was fixed their site popped back up in the SERPs and all was well and good.
While this type of event is not common, the robotx.txt file can harm your SEO if it is not used correctly.
Unlike a sitemap, which tells Google and other search engines about the content you want to be found and indexed. A Robots.txt file lists pages on your site that you don’t want to be crawled or show up in the SERPs.
The only reason that you would want to change your robots.txt file is if there are specific pages that you don’t want to be found. For example, if you have two pages with identical content on your website then you may want to consider blocking one of them to avoid duplicate content issues. By disallowing this page you are telling specific search engines bots that you don’t want the page to be crawled by the spiders.
While a robots.txt file can help you avoid having certain pages on your website indexed, there are some bots that will crawl the pages anyway.
For this reason, you may want to keep pages from being indexed by using noindex or nofollow tags. When you add these to your pages it adds an additional layer of security to specific pages to keep them from being crawled and appearing in the SERPs.
Checking to make sure that your robots.txt file is set up correctly can be a good thing but you should only make changes that are absolutely necessary.
If you play around with it too much you may end up with a site that no longer exists in the commanding eyes of Google.