Should you use Robots.txt or Meta Robots to control how spiders access your website? 

Posted on in Uncategorized

seo-1287369-mControlling how spiders access your website can be easily overlooked. However, it is important in terms of your site’s search engine optimization. Both Robots.txt and Meta Robots are used to control how spiders access your website. However, they are not equal in terms of how they accomplish this task. Which one should you use?

While Robots.txt can be used to control access to your website, current SEO best practice calls for you to use Meta Robots. For example, when you are seeking to block search engines from accessing a given page, you can block with robots.txt or Meta NoIndex.

Blocking with Robots.txt tells search engines not to crawl the page, but they can index it and display it in the results. Blocking with Meta NoIndex tells search engines they can visit the page, but they cannot display it in the results. Therefore, MetaNoIndex offers the most favorable result.

Additionally, pages using Robots.txt cannot reap the benefits of popularity and trust that comes from accumulating links since they do not allow search engines to crawl them. On the other hand, with meta robots, these pages can be crawled by search engines and their ranking is benefited by their popularity and trust.

While the differences between Robots.txt and Meta Robots may seem small to a lay person, the impact on SEO between the two can be significant. If you’re not sure how you currently are controlling access to your website, you’ll want to talk with your web designer sooner rather than later. It could be impacting your site’s page ranking in the search engine results pages (SERPs).

What our clients are saying

I want to thank you for your help with our website development project. Your programming skills, attention to detail, and ability to interact directly with our client were excellent. You have provided tremendous value to my organization. With a project of this magnitude, including the use of web services to push data out to 19 sub-websites, your ability to overcome our unique challenges and coordinate with my IT team was exceptional. I appreciate your help, and I am certain that your expertise was key to the project's success. Thanks again, I look forward to working with you on our next endeavor!

~ John Forgione – Executive Vice President – SFA Marketing, Inc.

Read More