Whenever I take over a website to help with improving the organic rankings, I always take a look at what Google has indexed and is currently displaying in search results for the website. In almost all case, if it’s a WordPress site, the search will show the theme folder has been indexed along with all the files in wp-content, wp-admin, and cgi-bin folder. Needless to say, Google, MSN, Yahoo, etc. has now indexed redundant files that have nothing to do with your website and in most cases this opens up possible security issues and can cause a hit to your organic search results by indexing these files.
The sad part is that all this can be prevented by the use of a good Robots.txt file that you can upload right to the root directory of your website. Takes a few minutes to do, and then you have to wait for the search engines to update your website indexing.
How to make a Great Robots.txt File.
