So I've been wondering why Google doesn't setup a URL that weblog software can "ping" each time a new entry is posted. I already ping and each time a new entry is posted. Why not rig up Google to do nearly real-time weblog indexing?

Now, I already know from my own stats that Google crawls my blog daily. Certainly we can improve on that.

Why does this matter? Simple. Virtually all of the traffic coming to my weblog that is not the result of someone else pointing at my weblog is from a Google search. I'd guess that Google generates 80-90% of my non-directly-linked hits. (Yahoo is a distant second place.)

Sure, Dave has suggested that Google use the info available at to do this, but why not eliminate the middle-man entirely? (No offense, Dave.)

Hmm. This gives me an idea for work that's sorta related to another idea I had after a co-worker showed me something that is best described as the opposite of Google Sets. Well, sort of. Lots of fun stuff to hack on there.

Posted by jzawodn at December 21, 2002 02:11 PM

Reader Comments
# jim winstead said:

or dave could implement the cloud interface (, and google and other services like technorati and (and could get real-time updates from instead of polling hourly. could implement it, too, and then people could ping whoever they want, and the various services would just share the data instead of people having to ping multiple services.

but why might google not be eager to rush into this? the seo crowd. this seems like it would be a ripe target for them to try abusing, so i expect any steps google takes to be measured.

on December 21, 2002 02:56 PM
# kasia said:

With movabletype it's a supporters thing.. If you donate money they give you a code that allows pinging their site.

on December 21, 2002 04:47 PM
# Dave Sifry said:

Well, I'd love to see this happen. I've offered Sergei Brin to come in and help them do it, but so far, no response.

on December 21, 2002 08:23 PM
# Dave Sifry said:

I meant, I offered TO Sergei that I'd come in and help them do it...

on December 21, 2002 08:31 PM
# Olivier Travers said:

In fact, I'm pretty sure this is going to be one of the major developments next year. Robots.txt is only a restriction list. Meta tags are useless. Google will probably want webmasters to adopt some kind of format or procedure to keep them posted about updates. They'll need to verify that the system is not abused to attract attention on sites that weren't really updated, but this would potentially allow them to keep a fresher index with less computing resources.

Let's take a hint from Froogle, their new shopping engine: they're asking merchants to provide them data feeds (from once a day to once a month.) They're most probably pondering how to do it most efficiently from a technical standpoint, but they might also have business model questions.

Google doesn't do paid inclusion, but what about paid refresh, i.e. it's free to be in the monthly Google dance, but daily refreshes are paying?

on December 22, 2002 07:19 AM
Disclaimer: The opinions expressed here are mine and mine alone. My current, past, or previous employers are not responsible for what I write here, the comments left by others, or the photos I may share. If you have questions, please contact me. Also, I am not a journalist or reporter. Don't "pitch" me.


Privacy: I do not share or publish the email addresses or IP addresses of anyone posting a comment here without consent. However, I do reserve the right to remove comments that are spammy, off-topic, or otherwise unsuitable based on my comment policy. In a few cases, I may leave spammy comments but remove any URLs they contain.