1
$\begingroup$

Suppose I have a Count time series data for the number of tasks that a server gets during a unit of time. Collecting data over a few months, I will get a dataset which will have 2 parameters.

First one will be the number of tasks which will be the dependent variable, and the next one will be the time (adjusted to 0), which will be the independent variable.

Now, as I understand it, the linear regression model cannot be used, as the relation between the number of tasks and time might not be linear. Can someone point out to the proper model for this kind of data?

$\endgroup$
2
$\begingroup$

You don't discuss a couple of key things: how wide your unit of time is (e.g., seconds, minutes, hours, days, etc.) or how many tasks occur per unit of time (e.g., handfuls, hundreds, thousands, etc). This will make a difference in terms of the functional form of the model.

Linear or not, integer counts of tasks as your dependent variable is likely to focus the modeling framework on one of a class of models called limited dependent variable models. These would include data generating processes such as the poisson model, the negative binomial model as well as zero-inflated models. One caveat to these approaches, however, is the question concerning the sheer quantity of tasks per unit of time. If the count of tasks is very large, e.g., hundreds or thousands of tasks, then you might want to consider some type of truncated (at zero) generalized linear (or nonlinear) model -- another flavor of limited DV models.

Let's assume that your event counts aren't huge. This would suggest a poisson process where the probability of an event occurring over some interval of time is proportional to the size of the interval, assuming the events are iid. The overriding assumption of the poisson model is that the variance equal the mean or, alternatively, that the variance is proportional to the mean.

There are a few considerations that can impact the choice of the poisson model: underdispersion (an infrequent result) or overdispersion. Overdispersion occurs when the variance is greater than the mean. There are lots of heuristics for identifying overdispersion in your data, e.g., the ratio of the std deviation to the mean, but formal tests of significance for overdispersion are also available to answer this question: e.g., see http://data.princeton.edu/wws509/notes/c4a.pdf (among others).

Overdispersion in the std errors suggests the negative binomial model which adds a multiplicative random effect θ to the underlying poisson model.

Another consideration involves the frequency with which your data process returns zero tasks or counts for your server per unit of time. If the poisson or negative binomial models are underestimating these zero-event periods, then you may want to shift to one of the zero-inflated approaches in modeling your information.

Classic Box-Jenkins-type approaches to modeling continuously distributed dependent variables would stipulate that the residuals from the model be HAC (heteroscedasticity and auto-correlation consistent). However, B-J models aren't appropriate for count DVs. Rob Hyndman discusses using a gamma function for modeling effects for trends or seasonality with count data here: Time series for count data, with counts < 20

Please note that you can fit additional predictors beyond just a single independent variable for time! Can the "tasks" be grouped or classified in some way? This would provide an additional explanatory variable. Is there any periodicity to the DV? Perhaps some time-based polynomials would improve the model fit. Have you explored lag functions? Etc., etc. Following on Hyndman's suggestions, trends can be linear and deterministic, nonlinear in the variables or nonlinear in the parameters. Polynomials can be used to fit any systematic periodicities that are nonlinear in the variables. Seasonality in your case would be dependent on the unit of time: e.g., if hourly then hourly dummies could be added as predictors. Dummies for the day of the week could also be added, as could weekly, monthly, quarterly, yearly dummies, and so on, up to the point at which they cancel each other out. Do you really have only one server or are there many servers? If so, then pooling the data across servers would help shrink, penalize or adjust the std errors. Gompertz growth models are one type of nonlinear in the parameters model for fitting exponential growth in tasks. They can be adapted for count data and also have a nice advantage insofar as they fit a sigmoid or S-curve to growth, which is consistent with an assumption that growth never, ever simply increases linearly out into the future but will have asymptotic ceilings, thresholds or limits. However, if your growth is super-exponential, as can occur in financial bubbles, then this requires a completely different approach which can't be developed within the context of this response.

Finally, poisson processes are agnostic as to the timing of when the next event is likely to occur. They only give you a probability that an event is likely to occur. Queuing theory as used in Operations Research would provide a framework for the timing of the next task. While I can point to the possibility of leveraging this class of models, queuing is not a spike for me in any way. Here is a reference for Basic Queuing Theory that may help: http://irh.inf.unideb.hu/~jsztrik/education/16/SOR_Main_Angol.pdf

Frequently, predicting the timing of a future event involves leveraging one of the Erlang distributions. However, I don't know if this distribution is appropriate for integer data.

$\endgroup$
  • $\begingroup$ Thanks a lot for the detailed answer. The material that i have read so far, also lead me to believe that Poisson model is the best for this kind of data. To clear out few things, the unit of time will be seconds, and the number of tasks arriving will be a handfull. The data can be grouped in different groups, but i don't want to make it more complicate then it is. $\endgroup$ – Haris Oct 25 '15 at 13:04
  • $\begingroup$ One more thing, i studied the Poisson model, but i really did not get a grip of it in practical terms. Can you please point me somewhere where i can find a real data example, with how alpha and beta are calculated. From beginning to end, the whole procedure. $\endgroup$ – Haris Oct 25 '15 at 13:06
  • $\begingroup$ Actually, there are lots of suggestions in the Cross-Validated side bars (see above and to the right). One discussion from a few years ago on airplane crashes does an excellent job of covering the waterfront... stats.stackexchange.com/questions/109390/… $\endgroup$ – Mike Hunter Oct 25 '15 at 13:51

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.