Robot txt files

[Koz]

Hi all,

I read this on another form. Have you heard of this before? Does anyone implement this?

If I'm not mistaken Lee wrote either here or on ForumTrends.com that he actually blocks all SE's via his robot.txt file from his forum and points them only to the forum archive so the risk of dup content isn't there.

Anyone else do this?

[Paz]

Brett Tabke famously blocked the spiders from his Webmasterworld forum, and lifted it very shortly afterwards:
http://blog.searchenginewatch.com/blog/051222-113443

I think the SE's have already figured out how archiving works in vBulletin forums. I don't see any evidence of a duplicate content penalty in my favourite SEO forums.....