Well, it seems that they're crawling now.

They don't appear to have hit my site(s) yet. But they're probably just starting to ramp up.

According to Feedster there aren't many bloggers talking about it yet. Anyone seen this thing hit their site yet? How long has it been coming? Does it visit your blog frequently (like Googlebot does)?

Posted by jzawodn at June 17, 2003 10:14 PM

Reader Comments
# Charles said:

My site doesn't have a robots.txt but you can be damn sure I am going to have one now:

User-agent: MSNBOT
Disallow: /

Thanks for the heads-up.

on June 17, 2003 10:56 PM
# BenM said:

any particular reason you want to disallow the microsoft bot from your site, other than it being from microsoft? I think that's a short sited attitude to take.

on June 18, 2003 01:23 AM
# Aaron Brazell said:

No, nope. No MSNBot at our site. Interesting though. I'll be watching for it.

on June 18, 2003 05:46 AM
# Luke Reeves said:

This bot started hitting my site a couple days ago (on the 16th) and has pulled pretty much every page I have.

It only crawls each page once, doesn't repeat them ala GoogleBot.

on June 18, 2003 06:10 AM
# Jeremy C. Wright said:

Googlebot's banging us, and a few others are as well, but the new MSN one isn't yet.

on June 18, 2003 07:06 AM
# Josh Woodward said:

I was actually just thinking of posting about the increased number of MSN Search hits I've been seeing. Specifically, they seem to pick up my blog entries with frightening speed (the next day usually). I don't know if this is at all related, since I haven't seen many of their bot hits, but it's interesting nonetheless.

on June 18, 2003 07:57 AM
# Jeremy Zawodny said:

MSN Search is current powered by Inktomi, which has some good built-in smarts about blogs. :-)

on June 18, 2003 08:04 AM
# Dan Kalowsky said:

First hit on or about April 28th 2003.

http://www.deadmime.org/~dank/archives/2003/04/26/Future Crawlers.html

on June 18, 2003 08:51 AM
# Charles said:

Do I need a reason other than it's Microsoft? I don't use Microsoft products, and I don't see any reason to allow Microsoft to use MY products.

on June 18, 2003 09:08 AM
# Anil said:

Do I need a reason other than it's Microsoft? I don't use Microsoft products, and I don't see any reason to allow Microsoft to use MY products.

Presuming that they eventually use this for MSN Search, do you see a reason for people who *do* use MS products to be able to find your site?

on June 18, 2003 09:37 AM
# Charles said:

That's specifically what I want to exclude. I will not permit my content to be repackaged and sold by Microsoft, in any format. Period. Sure I'll miss some hits from MSN lusers. Screw em.

on June 18, 2003 10:08 AM
# Dave Winer said:

An anonymous source provides details on MSNBot. Google in the crosshairs. MS offered to buy Google and was turned down. Search made $150 million profite for MS last year. Will be baked into Longhorn.

on June 18, 2003 10:17 AM
# Kevin said:

I noticed it show up a few weeks ago. It seemed pretty well behaved, ignoring the stuff it should ignore and scanning at a reasonable pace. No idea how good the search results are but the actual bot seems pretty reasonable.

on June 18, 2003 10:24 AM
# Jon Gales said:

Haven't seen it at any of my sites yet... MobileTracker gets a LOT of msn search referrals, so hopefully MSNBOT will be as kind as the Inktomi. :)

on June 18, 2003 10:25 AM
# Patrick Berry said:

It hit for the first time last night. 2003-06-17 21:52:44 was the first hit for robots.txt. 35 more hits at varying intervals of anywhere from 1 minute to 6 hours. It hasn't looked anywhere it isn't supposed to, not that I expected it to...

on June 18, 2003 10:50 AM
# Dave Winer said:

Here's Microsoft's about page for MSNBot.


on June 18, 2003 12:47 PM
# pb said:

You don't need another reason other than that it's Microsoft unless you're trying to appear rational.

on June 18, 2003 12:51 PM
# Tristan Louis said:

It visited my site (just checked the logs) today but hasn't been so intrusive to date (it's on the low end of the list in terms of user agents)

on June 18, 2003 01:58 PM
# Joe Banks said:

Well I certainly hope MS search is going to be developed because it's been a total joke for years. The MSDN site in particular seems to have the worst search results -- worse than search engines that came out circa 1995. Most times, even if you know exactly what you looking for and add several keywords, it misses it entirely. I don't even bother with it and just use google and a -site restriction. Hopefully they'll get with the frickin' program.

on June 18, 2003 04:14 PM
# Charles said:

he he.. people griping about "appearing rational" in a thread with Dave Winer in it.. ROTFLMAO!

on June 18, 2003 05:23 PM
# Fruetel said:

That plans of Microsoft are scary stuff IMHO. I donīt like the quasi monopoly of Google we have now, but a Microsoft monopoly on web searches would be even worse. This company allready has so much power that most people canīt really choose what OS they should be working with. If one day they have the power to dictate people what websites to visit, the internet might become a pretty useless place.

on June 20, 2003 12:11 PM
# eLGie said:

Inktomi Slurp it up baby. I don't mind the MSNBot crawling and it has for the last three months. I get so many referrals from that engine that I wonder if google is all that it is. Nice to see the competition come in from somewhere now if only there was a third engine out there that would catch on big I would love it.

on June 22, 2003 07:05 PM
# Nono said:


on February 3, 2005 02:36 AM
# Nono said:


on February 3, 2005 02:36 AM
# Nono said:


on February 3, 2005 02:36 AM
# Nono said:


on February 3, 2005 02:36 AM
Disclaimer: The opinions expressed here are mine and mine alone. My current, past, or previous employers are not responsible for what I write here, the comments left by others, or the photos I may share. If you have questions, please contact me. Also, I am not a journalist or reporter. Don't "pitch" me.


Privacy: I do not share or publish the email addresses or IP addresses of anyone posting a comment here without consent. However, I do reserve the right to remove comments that are spammy, off-topic, or otherwise unsuitable based on my comment policy. In a few cases, I may leave spammy comments but remove any URLs they contain.