# Web page change frequency

I want to know I often I need to crawl certain webpages, to make sure I have the newest content as fast as possible. The naïve approach to this, is to simply calculate the average over a certain period of time. However, different web pages are updated at different rates, so this wouldn't be a good fit. I did some googling, and apparently, the Poisson distribution is a good fit for this kind of estimation. My question is, how should I calculate lambda? And it should be constantly updated, correct?

Thank you

-
Why the Poisson? That is used for modelling counts. –  Peter Flom Oct 17 '12 at 10:14

2. Use the estimate of $\lambda$ from the model to scholastically decide whether or not to check for an update right now, based on the time since the last update
then, you can estimate $\lambda$ for a given webpage as $\sum_{k\in K} \frac{t_k}{|K|}$, where $K$ is the set of all updates you've seen on the page, and $t_k$ is the length of time that passed between update $k$ and immediately preceding update.
Of course, if $\lambda$ drifts with time, or the frequencies of updates are conditionally related to each other (which is possible for certain types of sites), then this might not be very useful. It does seem to be a standard technique for this kind of thing though, so perhaps it will be fine.