3
$\begingroup$

I am attempting to predict how many times a certain event will happen in a time period. For instance, predict the number of time the event will occur in the next 5 hours. I have data going back a couple years just listing the time the event occurred. Here is a graph of the total number of events vs time. I also have external variables related to the event, such as time of day or weather, which I would like to incorporate into the model.

Event time period Shorter time period So far, I've tried a linear regression algorithm, but proved to not be very accurate, likely due to the overall increasing trend. I also tried a neural net regression technique with a dataset consisting of the time until the end of the time period I'm predicting, the number of occurrences so far, and the total number of occurrences I am trying to predict, but it ended up just memorizing the dataset and not actually learning anything. Any help much appreciated!

$\endgroup$
1
$\begingroup$

The number of counts in a specific time period is modeled using a counting process with a deterministic or stochastic intensity. Stochastic process $N(t)$ is a counting process (point process) if

  • $N(t)$ takes non-negative integer values,
  • in each random scenario, the trajectory of $N(t)$ is piece-wise constant,
  • in each random scenario, the trajectory of $N(t)$ is right-continuous,
  • $N(t)$ is non-decreasing.

The intensity of counting process $N(t)$ is defined to be a stochastic process $\lambda(t)$ such that, for any time $t$ and shift $u$, $$ P(N(t+u) - N(t) = 1 | N(s), 0 \leq s \leq t) = \lambda(t) u + o(u), $$ where $o(u)$ is a function of $u$ which converges to $0$ when $u$ converges to $0$. The intensity stands for the expected number of jumps per unit of time in a very small time interval.

Counting process is a generalization of a non-homogeneous Poisson process. Non-homogeneous Poisson process jumps with a deterministic, time-varying intensity while counting process jumps with a stochastic, time-varying intensity (generally speaking). Typically, diffusion processes are used to model the intensity $\lambda(t)$. Ito diffusion is a stochastic process $X(t)$ satisfying the following stochastic differential equation: $$ dX(t) = A(t,X(t)) dt + B(t,X(t)) dW(t), $$ where $W(t)$ is Brownian motion. In particular, if we choose to employ affine diffusions the counting process gains certain nice properties. In affine diffusions, $A(t,X(t))$ and $B(t,X(t))^2$ can be written as linear functions of $X(t)$.

This approach is the industry standard in actuarial science, derivatives pricing and survival analysis. If you have seen some simpler models, the chances are they fall within the aforementioned framework as special cases.

NOTES (addressing the follow-up questions):

1] $o(u)$ stands for "small o". The term is part of calculus. By definition, function $g(u)$ is "small o" with respect to function $f(u)$ when $u$ converges to $a$ if $g(u) / f(u)$ converges to 0 when $u$ converges to $a$.

2] Modeling jump intensity $\lambda(t)$ as a deterministic or stochastic process is a big field. Many methods and modeling ideas are part of it. $\lambda(t)$ does not always have to be complex but, in the presence of substantial amount of data, a clever diffusion-based model may improve the goodness of fit... To get acquainted with the field, please refer to the following references:

  • Duffie, D., & Singleton, K. (2003). Credit Risk: Pricing, Measurement, and Management. Princeton University Press.

  • Rausand, M. & Høyland, A. (2004). System Reliability Theory: Models, Statistical Methods, and Applications. Wiley-Interscience, Hoboken, New Jersey.

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ Thank you for the detailed response! I do have a couple questions. First off, what does the function o represent? Also, how would I figure out the λ equation for my data? You mention using a diffusion process such as Ito diffusion. How would I use my data to implement such a process? My apologies if these questions seem ignorant, which I am in this subject quite frankly. $\endgroup$ – Anthony Ebiner Nov 1 '19 at 21:55
  • $\begingroup$ @AnthonyEbiner I have edited my response to address your questions. Please see above. $\endgroup$ – stans - Reinstate Monica Nov 2 '19 at 1:24
1
$\begingroup$

A possibly simpler approach to a full-blown stochastic process would be to use standard time series forecasting tools. Bin your events in an appropriate time granularity, e.g., in hourly buckets, or even in 5-hour buckets. Then use a standard forecasting algorithm, such as exponential smoothing. I very much recommend this open online textbook.

Be aware that even the best prediction models may not reach the level of accuracy you would like to see. How to know that your machine learning problem is hopeless? may be helpful.

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ Thank you for the response and textbook recommendation, it looks like it will be very helpful. You mention exponential smoothing, and when I look at examples of its use, I see cases of it predicting a value associated with each time value, such as sales or price, which can go up or down. Since I am hoping to forecast the total number of events, a value that can only go up, could I still use this algorithm? Also, if I were to bin the events in hourly buckets, would I have each bin's value set to the total number of events up until that time, or the number of events that occurred during hour? $\endgroup$ – Anthony Ebiner Nov 1 '19 at 21:40
  • 1
    $\begingroup$ Two possibilities. (1) Model total events up to a regular time point, e.g., up to each time point. Any forecasting algorithm should note the positive trend and extrapolate it. Or (2) model the incremental events happening in each time bucket, then forecast these out, and aggregate the forecasts over time. Which approach is better can't be said beforehand. $\endgroup$ – Stephan Kolassa Nov 1 '19 at 21:48
  • 1
    $\begingroup$ Yet better: do both (1) and (2) and average the resulting forecasts. I suspect the result of this may be better yet than either one in isolation. $\endgroup$ – Stephan Kolassa Nov 1 '19 at 21:49
  • $\begingroup$ That makes perfect sense, thanks! $\endgroup$ – Anthony Ebiner Nov 1 '19 at 21:57

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.