cliveportman/code

How often does Google update its index?

  1. Google appears to cache pages around a similar date. Checking out the cached dates of various creative sites in Norwich, they were all the 8th, 9th or 10th of October. That was on 15th or 16th October. I expanded the search outside Norwich and found a similar result. When I looked at different sector sites, I started to see more variation.
  2. Checking those same cached pages on 18th October, the dates are still clustered, but the dates are now 12th and 13th October. Google is holding on to cached pages for several days before publishing them.
  3. Webmaster Tools shows Google crawls individual sites every day.
  4. A Fetch as Googlebot submission can result in significant index changes within a couple of hours. I've only seen this twice, though.
  5. Typically, a Fetch as Googlebot submission does not prompt index changes for several days, if at all.
  6. Significant index changes due to onsite improvements occur every couple of weeks. This comes from me comparing ranking data with changes to websites.
  7. The cache and the index are not in-sync. The index may link to a cached page but return a 404 error.
  8. The index is updated 3 or 4 days after the cache.

I can't believe the index would a) be crawled separately to the cache, nor b) not use the same data. Which leads to a hypothesis of:

Google crawls, then several days later, updates the cache with that crawl data, then, several days later, updates the index with that same data.

Some questions that raises: