crawler directives
Tag details
Welcome to the 'crawler directives' tag page at Technorati. This page features content from the farthest reaches of the Blogosphere that authors have "tagged" with 'crawler directives'.
Latest blogosphere posts tagged “crawler directives”
-
The perfect robots.txt for News Corp
Sebastian's Pamphlets —
Authority: 508
I appreciate Google’s brand new News User Agent . It is, however, not a perfect solution, because it doesn’t distinguish indexing and crawling . Disallow is a crawler directive, that simply tells web robots “do not fetch my content”. It doesn’t prevent contents from indexing. That means, search ...1 week ago
