You're blocking Yahoo! Search crawlers from your site and have been for several days now. We've tried contacting you to verify your intent. Most folks would just add the relevant entries to a robots.txt file, so we're thinking maybe it's a bug in your ACLs somewhere.

Anyway, if you'd like to stay in the search index please contact me and I'll get you in touch with someone on our site who can help sort it out if need be.

If you really do want us out, how about just using robots.txt so we're clear about what's going on?


Update: Thanks for the leads. We got a contact.

Posted by jzawodn at January 18, 2006 09:08 PM

Reader Comments
# Robert Oschler said:


Being a SourceForge member I just e-mailed the people I know at SourceForge, VASoftware, and The Open Source Technology Group about your blog post. I'm sure someone will get back to you soon.

My project was Robosapien Dance Machine, the SourceForge Project Of The Month for May 2005. Yeah it's a shameless plug. It's late at night, I'm shameless, and I plugged it! Besides it's free. :)

Home Page -
Funny Robot Burping Contest Video -

on January 18, 2006 10:38 PM
# James Day said:

When I and others at WIkipedia have blocked search engine crawlers the usual cause was indirect crawling via a third party site proxying to us, with that third party site not having a suitable robots.txt file to protect things which needed protecting and not having a suitable Crawl-delay value.

Crawl-delay doesn't seem to be sensitive enough with only a 1 second apparent finest resolution and a million plus pages to crawl in a month or so. Would be nice to say number of concurrent crawlers and page delay per crawler, which could perhaps be in the 100ms range. Specifying "off peak" and "on peak" times and rate multipliers might also be helpful - could offer ten or more times the crawl rate at off peak times.

on January 18, 2006 11:47 PM
# Paul Querna said:

Good luck getting SourceForge to unblock you. They have been blocking the Blogliens crawler for months, and we haven't been able to get any agreement on working around the issue.

on January 19, 2006 12:40 AM
# Um said:
on January 19, 2006 01:43 AM
# Jeff Barr said:

If you get to talk to them, you may want to tell them that their "daily updates" mail has been listing the same applications as "new" for the last 5 days.

on January 19, 2006 04:24 PM
# Robert Oschler said:


I just got a reply e-mail back from SourceForge HQ thanking me for the notice and saying that they will be looking in to this in the next few days. Let me know if things get worked out or not.


on January 20, 2006 06:38 AM
# said:

IMO, blocking Yahoo is a good thing. Encourages people to use a decent search engine, or should I say THE decent search engine.

on January 20, 2006 09:03 AM
# Robert Oschler said:


>>IMO, blocking Yahoo is a good thing. Encourages people to use a decent search engine, or should I say THE decent search engine.

Have you given their search a fair open minded try latetly? Yahoo Search 2006 is nothing like Yahoo Search of old. In several cases, for example, they have almost 3 times as many indexed pages. The search results are solid and their Local Search is fantastic. In addition they have creative tools like Mindset which allows you to tune your search results between commercial and non-commercial results. Have you at least tried their creative commons search?

I am not being an acolyte here or challenging you, I just want you to get the best search experience possible. BTW, I *don't* work for Yahoo!; not even as a consultant.

on January 20, 2006 09:41 AM
Disclaimer: The opinions expressed here are mine and mine alone. My current, past, or previous employers are not responsible for what I write here, the comments left by others, or the photos I may share. If you have questions, please contact me. Also, I am not a journalist or reporter. Don't "pitch" me.


Privacy: I do not share or publish the email addresses or IP addresses of anyone posting a comment here without consent. However, I do reserve the right to remove comments that are spammy, off-topic, or otherwise unsuitable based on my comment policy. In a few cases, I may leave spammy comments but remove any URLs they contain.