Well, it seems that they're crawling now.
They don't appear to have hit my site(s) yet. But they're probably just starting to ramp up.
According to Feedster there aren't many bloggers talking about it yet. Anyone seen this thing hit their site yet? How long has it been coming? Does it visit your blog frequently (like Googlebot does)?
Posted by jzawodn at June 17, 2003 10:14 PM
My site doesn't have a robots.txt but you can be damn sure I am going to have one now:
User-agent: MSNBOT
Disallow: /
Thanks for the heads-up.
Charles,
any particular reason you want to disallow the microsoft bot from your site, other than it being from microsoft? I think that's a short sited attitude to take.
No, nope. No MSNBot at our site. Interesting though. I'll be watching for it.
This bot started hitting my site a couple days ago (on the 16th) and has pulled pretty much every page I have.
It only crawls each page once, doesn't repeat them ala GoogleBot.
Googlebot's banging us, and a few others are as well, but the new MSN one isn't yet.
I was actually just thinking of posting about the increased number of MSN Search hits I've been seeing. Specifically, they seem to pick up my blog entries with frightening speed (the next day usually). I don't know if this is at all related, since I haven't seen many of their bot hits, but it's interesting nonetheless.
MSN Search is current powered by Inktomi, which has some good built-in smarts about blogs. :-)
First hit on or about April 28th 2003.
http://www.deadmime.org/~dank/archives/2003/04/26/Future Crawlers.html
Do I need a reason other than it's Microsoft? I don't use Microsoft products, and I don't see any reason to allow Microsoft to use MY products.
Do I need a reason other than it's Microsoft? I don't use Microsoft products, and I don't see any reason to allow Microsoft to use MY products.
Presuming that they eventually use this for MSN Search, do you see a reason for people who *do* use MS products to be able to find your site?
That's specifically what I want to exclude. I will not permit my content to be repackaged and sold by Microsoft, in any format. Period. Sure I'll miss some hits from MSN lusers. Screw em.
An anonymous source provides details on MSNBot. Google in the crosshairs. MS offered to buy Google and was turned down. Search made $150 million profite for MS last year. Will be baked into Longhorn.
I noticed it show up a few weeks ago. It seemed pretty well behaved, ignoring the stuff it should ignore and scanning at a reasonable pace. No idea how good the search results are but the actual bot seems pretty reasonable.
Haven't seen it at any of my sites yet... MobileTracker gets a LOT of msn search referrals, so hopefully MSNBOT will be as kind as the Inktomi. :)
It hit for the first time last night. 2003-06-17 21:52:44 was the first hit for robots.txt. 35 more hits at varying intervals of anywhere from 1 minute to 6 hours. It hasn't looked anywhere it isn't supposed to, not that I expected it to...
You don't need another reason other than that it's Microsoft unless you're trying to appear rational.
It visited my site (just checked the logs) today but hasn't been so intrusive to date (it's on the low end of the list in terms of user agents)
Well I certainly hope MS search is going to be developed because it's been a total joke for years. The MSDN site in particular seems to have the worst search results -- worse than search engines that came out circa 1995. Most times, even if you know exactly what you looking for and add several keywords, it misses it entirely. I don't even bother with it and just use google and a -site restriction. Hopefully they'll get with the frickin' program.
he he.. people griping about "appearing rational" in a thread with Dave Winer in it.. ROTFLMAO!
That plans of Microsoft are scary stuff IMHO. I donīt like the quasi monopoly of Google we have now, but a Microsoft monopoly on web searches would be even worse. This company allready has so much power that most people canīt really choose what OS they should be working with. If one day they have the power to dictate people what websites to visit, the internet might become a pretty useless place.
Inktomi Slurp it up baby. I don't mind the MSNBot crawling and it has for the last three months. I get so many referrals from that engine that I wonder if google is all that it is. Nice to see the competition come in from somewhere now if only there was a third engine out there that would catch on big I would love it.
