Time series forecasting using statistical tools

Question

I'm building a system which needs to poll some feed of articles in a smart way. When polling, I can only know the number of new articles (could be $0$ - no new articles). I don't have the info when each article was published.

So I thought about a relatively simple solution: Exponential Moving Average. Something like: $$t_n = \alpha\cdot t_{n-1} + (1-\alpha)\cdot p$$

Where $p$ is the time difference between the last poll to the last successful poll (so if there are no new articles, $p$ is getting larger and larger and so $t_n$ is).

Requests and Thoughts

1. I'd like to get a critique on the above formula (Would you define it differently?)
2. How can I define $\alpha$ efficiently? Could it be dynamic/changing over time?
3. I've read about more sophisticated statistical tools (like ARMA/ARIMA) for forecasting time series. Most of them uses the errors (the time difference between the forecast time and the actual time), but I don't own this information, sadly. What statistical model fits my scenario?
4. Currently, I don't use the number of articles, though they could be useful to evaluate the next poll.

Thanks!

---

P.S.
I haven't checked it but I think the behavior of feeds is somewhat trendy/seasonal (For example, fewer articles at night time). On the other hand, there are what is called if I'm not mistaken, "random shocks" (For example, a terror attack).

Answer 1

Apparently the only information you have is the numbers of days and the number of news articles published on the feed.

I think what you are really asking is "Is this news feed worth my effort to poll?" and the desired answer is "Yes or No".

Therefore, you are actually wanting to perform a logistic regression. The result of logistic regression is a probability and in your case, the probability of whether you should poll the news feed. After you make your model, you then need to decide your threshold for action: Perhaps for Feed A (which is really important), you want to poll it if the probability is > 75% but for Feed B (maybe it is not so important) you may decide to have a higher threshold, maybe poll it only if the probability of a new feed is > 90%.

In your case, you have one additional component - time dependent data that you think are involved in this whole process.

I would suggest creating moving window logistic regression (like this example for linear regression). You of course would have to tune the number of days you want to incorporate on your model based on some modeling you would do ahead of time, and of course you would have to evaluate your model periodically!

Answer 2

Please don't get upset, I'm a newbie. My idea is even simpler than yours.

So you have to poll in a smart way, that is when the probability there are news articles on the feed is higher. In my opinion the point is: "What is the probability to get new articles from $feed_n$ if today I poll it ?", if the probability is little, we don't waste time polling.

I'm thinking Poisson distribution. In the begin you have to poll each feed using the same frequency, once you get enough data you can start using it.You can also update the feeds models each month using the data you collected.

Most likely there are correlations between your feeds, but as simple start thinking feeds indipendent is a good option.