B2B SEO - Marketing, Search Engine Optimization for Business

Entries Tagged as 'Search Engine Robots'

Robots Exclusion Standard

January 8th, 2007 · No Comments

The robots exclusion standard, also known as the Robots Exclusion Protocol or robots.txt protocol is a convention to prevent cooperating web spiders and other web robots from accessing all or part of a website which is, otherwise, publicly viewable. Robots are often used by search engines to categorize and archive web sites, or by webmasters […]

[

Tags: Search Engine Robots