I want to know I often I need to crawl certain webpages, to make sure I have the newest content as fast as possible. The naïve approach to this, is to simply calculate the average over a certain period of time. However, different web pages are updated at different rates, so this wouldn't be a good fit. I did some googling, and apparently, the Poisson distribution is a good fit for this kind of estimation. My question is, how should I calculate lambda? And it should be constantly updated, correct?
Edit: John's answer helped me out; for further reference, check this article: Estimating Frequency of Change