You're blocking Yahoo! Search crawlers from your site and have been for several days now. We've tried contacting you to verify your intent. Most folks would just add the relevant entries to a robots.txt file, so we're thinking maybe it's a bug in your ACLs somewhere.
Anyway, if you'd like to stay in the search index please contact me and I'll get you in touch with someone on our site who can help sort it out if need be.
If you really do want us out, how about just using robots.txt so we're clear about what's going on?
Thanks!
Update: Thanks for the leads. We got a contact.
Posted by jzawodn at January 18, 2006 09:08 PM
Jeremy,
Being a SourceForge member I just e-mailed the people I know at SourceForge, VASoftware, and The Open Source Technology Group about your blog post. I'm sure someone will get back to you soon.
My project was Robosapien Dance Machine, the SourceForge Project Of The Month for May 2005. Yeah it's a shameless plug. It's late at night, I'm shameless, and I plugged it! Besides it's free. :)
Home Page - http://www.robodance.com/
Funny Robot Burping Contest Video - http://www.robotsrule.com/html/robodance-preview.php
When I and others at WIkipedia have blocked search engine crawlers the usual cause was indirect crawling via a third party site proxying to us, with that third party site not having a suitable robots.txt file to protect things which needed protecting and not having a suitable Crawl-delay value.
Crawl-delay doesn't seem to be sensitive enough with only a 1 second apparent finest resolution and a million plus pages to crawl in a month or so. Would be nice to say number of concurrent crawlers and page delay per crawler, which could perhaps be in the 100ms range. Specifying "off peak" and "on peak" times and rate multipliers might also be helpful - could offer ten or more times the crawl rate at off peak times.
Good luck getting SourceForge to unblock you. They have been blocking the Blogliens crawler for months, and we haven't been able to get any agreement on working around the issue.
If you get to talk to them, you may want to tell them that their "daily updates" mail has been listing the same applications as "new" for the last 5 days.
Jeremy,
I just got a reply e-mail back from SourceForge HQ thanking me for the notice and saying that they will be looking in to this in the next few days. Let me know if things get worked out or not.
Robert
IMO, blocking Yahoo is a good thing. Encourages people to use a decent search engine, or should I say THE decent search engine.
Anonymous,
>>IMO, blocking Yahoo is a good thing. Encourages people to use a decent search engine, or should I say THE decent search engine.
Have you given their search a fair open minded try latetly? Yahoo Search 2006 is nothing like Yahoo Search of old. In several cases, for example Flickr.com, they have almost 3 times as many indexed pages. The search results are solid and their Local Search is fantastic. In addition they have creative tools like Mindset which allows you to tune your search results between commercial and non-commercial results. Have you at least tried their creative commons search?
I am not being an acolyte here or challenging you, I just want you to get the best search experience possible. BTW, I *don't* work for Yahoo!; not even as a consultant.
