Web page change frequency

Question

I want to know I often I need to crawl certain webpages, to make sure I have the newest content as fast as possible. The naïve approach to this, is to simply calculate the average over a certain period of time. However, different web pages are updated at different rates, so this wouldn't be a good fit. I did some googling, and apparently, the Poisson distribution is a good fit for this kind of estimation. My question is, how should I calculate lambda? And it should be constantly updated, correct?

Edit: John's answer helped me out; for further reference, check this article: Estimating Frequency of Change

Thank you

Answer 1

Assuming what you want to do is:

1. Build a model of the number of updates over a time interval t, under the assumption that the number of updates follows a Poisson distribution (reasonable assumption).

2. Use the estimate of $\lambda$ from the model to scholastically decide whether or not to check for an update right now, based on the time since the last update

then, you can estimate $\lambda$ for a given webpage as $\sum_{k\in K} \frac{t_k}{|K|}$, where $K$ is the set of all updates you've seen on the page, and $t_k$ is the length of time that passed between update $k$ and immediately preceding update.

Of course, if $\lambda$ drifts with time, or the frequencies of updates are conditionally related to each other (which is possible for certain types of sites), then this might not be very useful. It does seem to be a standard technique for this kind of thing though, so perhaps it will be fine.