1
$\begingroup$

I would like to detect outliers in real-time data that is aggregated per hour. For this example, I've selected the hourly pedestrian data from Melbourne, Australia (Pedestrian volume (updated monthly), Pedestrian Counting System)

I understand there are a large number of existing detection algorithms, which in time I'll learn and use. In the short term I'd like to use the simplest approach. One such method is outlined by @Aksakal in the following stackexchange post:

What algorithm should I use to detect anomalies on time-series?

I think the key is "unexpected" qualifier in your graph. In order to detect the unexpected you need to have an idea of what's expected.

I would start with a simple time series model such as AR(p) or ARMA(p,q). Fit it to data, add seasonality as appropriate. For instance, your SAR(1)(24) model could be: $y_{t}=c+\phi y_{t-1}+\Phi_{24}y_{t-24}+\Phi_{25}y_{t-25}+\varepsilon_t$, where $t$ is time in hours. So, you'd be predicting the graph for the next hour. Whenever the prediction error $e_t=y_t-\hat y_t$ is "too big" you throw an alert.

When you estimate the model you'll get the variance $\sigma_\varepsilon$ of the error $\varepsilon_t$. Depending on your distributional assumptions, such as normal, you can set the threshold based on the probability, such as $|e_t|<3\sigma_\varepsilon$ for 99.7% or one-sided $e_t>3\sigma_\varepsilon$.

The number of visitors is probably quite persistent, but super seasonal. It might work better to try seasonal dummies instead of the multiplicative seasonality, then you'd try ARMAX where X stands for exogenous variables, which could be anything like holiday dummy, hour dummies, weekend dummies etc.

Unfortunately the original post does not go into details, hence my question:

How is the variance $\sigma_\varepsilon$ of the ARIMA error term ($\varepsilon_t$) calculated for the case outlined above?

I could not find any good references on detailing this process. Penn Stat provides a good overview of using Linear Regression Models with Autoregressive Errors, but does not seem to cover this area.

Any guidance would be appreciated.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.