Dear MSNBot,

Take a chill pill. You've been pulling my RSS feed roughly every 10 minutes today.

I know you're late to the search game and even later to the RSS game, but seriously... WTF?!

Please take a lesson from the rest of the RSS aggregation and search services. Listen to a ping stream and think about checking once an hour or so.

Thanks,

Jeremy

Posted by jzawodn at August 18, 2005 05:53 PM

Reader Comments
# Glen Campbell said:

One of my servers has had 2,397 hits from msnbot today (as of 8pm).

on August 18, 2005 08:20 PM
# Chris DiBona said:

You folks aren't kidding, msnbot is on crank. Crawling more often does not equal a better crawl!

on August 18, 2005 08:40 PM
# Mark Beeson said:

This happened to my feed yesterday to the point where it was hitting it once every minute. It's still going today, but it looks like it has slowed. From looking through the logs, it looks like MSN is trying to determine the refresh rate of my feed (and ignoring a Last-Modified: header). I see it hitting more often during some hours, and then slowing, and now it seems as if it has settled on a constant once-every-five-minutes.

on August 18, 2005 09:14 PM
# MSNBot said:

Dear Jeremy,

Sorry about that. I will try to lay off the meth.

Sincerely,
MSNBot

on August 18, 2005 10:36 PM
# KCinDC said:

It's been grabbing my feed about every five minutes since Wednesday evening. And it doesn't ask for gzip compression to save bandwidth.

on August 19, 2005 02:22 PM
# Ron K. Jeffries said:

I am not on the A list like Jeremy, but I am
puzzled by how much crawing Google seems to do vs other spiders. I am new to this so I may be reading my stats wrong:
http://blog.eronj.com/awstats/awstats.pl

Robots/Spiders visitors
18 different robots Hits Bandwidth

Last visit
Googlebot 18562 569.74 MB 19 Aug 2005 - 01:19

MSNBot 1436 6.78 MB 19 Aug 2005 - 00:48

Inktomi Slurp 713 10.16 MB 19 Aug 2005 - 01:02

on August 19, 2005 02:33 PM
# indy said:

so adding this to your pages doesn't help?
User-agent: msnbot
Crawl-delay: a bazillion

on August 20, 2005 04:38 AM
# Sriram Krishnan said:

See http://blogs.msdn.com/msnsearch/archive/2005/08/22/454544.aspx

on August 22, 2005 09:58 AM
# Kim Stian Ervik said:

Ron K. Jeffries: Those Googlebot crawls are just insane for a site like yours.

on August 23, 2005 12:31 AM
# Irvin Owens Jr said:

I wonder if the MSN bot can read the sitemap.xml file, perhaps that would cool it off. I know MS said they fixed it, or were fixing it, but it still seems to be all over my site.

on August 26, 2005 03:10 PM
Disclaimer: The opinions expressed here are mine and mine alone. My current, past, or previous employers are not responsible for what I write here, the comments left by others, or the photos I may share. If you have questions, please contact me. Also, I am not a journalist or reporter. Don't "pitch" me.

 

Privacy: I do not share or publish the email addresses or IP addresses of anyone posting a comment here without consent. However, I do reserve the right to remove comments that are spammy, off-topic, or otherwise unsuitable based on my comment policy. In a few cases, I may leave spammy comments but remove any URLs they contain.