You have to marvel at how fast and powerful Google's technology has become.

Remember years ago, how long it took to get a site listed? Remeber waiting around for the monthly updates to see changes in SERPs.

A Techcrunch article debuted 26 minutes ago, and was spidered and incorporated in the organic SERPs, coming up for keywords in the title -
'Google Mafia'

http://www.freezepage.com/1198859116XDTQBARUSV

http://www.techcrunch.com/2007/12/28/here-comes-the-google-mafia/

I have noticed this thing with my Google alerts & blogs
Alert hardly takes 30 min to reach my inbox after i write a post in my blog or here at blogoscoped

Search-Engines-Web.com > I don't understand why you open this thread. This exist for months (Universal Search), and there are many example more explicits: a famous French sportswoman have problems recently (she appears naked on really really private pictures...) and I remember that I have seen a blog ranked first for her name, with the caption "6 minutes ago"...

hm, yes. i got some posts from my blog indexed in < 30 minutes...

http://www.freezepage.com/1198994882SNNJQKEFDY

This latest post was spidered and ranked in 15 minutes on the organic SERPs

and in 10 minutes on the blogsearch

This is how fast the technology is evolving

just imagine a couple of years from now – it will be done in almost real time on all the major search engines

I agree with Tom's comments. This is not new, but it's getting real faster. I am getting near practically instance info via the RSS blips within my clipbar. A year ago it was like 20 min for it to get aggregated into GDS and now its 1Min!!

  

Beyond mere technological restrictions of how often a spider can crawl the site, there's also the netiquette of "don't pull the same file too often". (I remember with RSS crawlers I set-up that 30 minutes was the way to go, and that if you polled the RSS too often it would return a blocked file for some sites.)
However, I'd love if important bots would spider say an RSS file or the homepage say every minute.
Then again do we want to give this right to every spider or just the Googlebot? If we give it just the Googlebot* (or just the major search engines) it may become an unfair netiquette, if we give it to every spider, the server load may increase quite a bit...

*E.g. via Google webmaster tools; at this time, the Tools -> "Set crawl rate" page exists, but the "Faster" value is ghosted (unavailable to choose)

Similar experience here, posts from my blog will be appear to the Google index in 15-16 minutes.

My blog posts take about 20 minutes to appear in Google SERPs which is odd seeing as Google doesn't crawl my site all that often suggesting that Google might be monitoring RSS feeds that it picks up via FeedFetcher and updating sites when its feed updates

Martin:
Yes, that's obvious and it's the best way to monitor when a site updates (dynamically-generated sitemaps are also good). The tricky part comes when you rank the page, but Google probably uses historical data about the site and changes the ranking after a number of minutes/hours when there's more information about backlinks.

The Google robot doesn't crawl the site itself daily, absolutely not, especially indexing Web pages updated very frequently Google watch RSS feeds.
BTW: List of some Googlebot UA strings:
http://www.useragentstring.com/pages/Googlebot/