0
$\begingroup$

Is anyone aware of machine learning models that are able to deal with heteroskedasticity in time series, when trying to detect outliers? There are a lot of anomaly detection tools out there (like k-means, isolation forests, one-class SVMs,...) but they do not really have the ability to deal with changes in volatility. Ideally, I would like to have an algorithm that looks locally at the time series. Otherwise, if there are periods in the data where there is high volatility, all of these observations will be flagged as outliers. However, compared to their neighbours these high volatility point should not all count as outliers. This happens because the methods known to me look at the problem globally. Taking a local perspective, observations in high volatility should not necessarily count as outliers. Are there any neural network representations that are able to cope with that? I heard about convolutional approaches that have a sliding window that they use to take a local perspective.

I could also imagine having to transform the data so that some of the outliers appear much more strongly to the mentioned techniques. However, I'm also not aware of any such transformations.

Edit

Here is what I have been describing. The red ones should be outliers, while the green ones should not be. What I'm looking for is an ML approach that allows for volatility clustering and local outlier detection. Because of the multiplicative nature of the underlying process, large jumps are likely clustered.

enter image description here

$\endgroup$
6
  • $\begingroup$ are your time series multivariate? $\endgroup$
    – carlo
    Commented Oct 29, 2021 at 10:17
  • $\begingroup$ Not necessarily. I would be glad to solve the univariate case first. So right now I'm only looking at a single time series. $\endgroup$
    – SimonDude
    Commented Oct 29, 2021 at 11:47
  • $\begingroup$ Can you show some examples of your data (or similar data), marked with what you consider to be outliers? That will help to understand the problem and to propose a suitable method $\endgroup$
    – Jon Nordby
    Commented Oct 29, 2021 at 21:24
  • $\begingroup$ Sorry for that, I added a figure. Just for reference, it's about financial time series data. $\endgroup$
    – SimonDude
    Commented Oct 30, 2021 at 17:36
  • $\begingroup$ That looks much better. What is the time resolution? Monthly/weekly/daily? $\endgroup$
    – Jon Nordby
    Commented Nov 12, 2021 at 11:59

1 Answer 1

1
$\begingroup$

You want to take into consideration some local patterns in time to define your anomalies. This can be done with a relatively simple feature engineering procedure, and then using the standard anomaly detection methods on top.

  1. Divide the time-series into sliding time-windows of a suitable length for the "context" you wish to have. In your example, this could be some months.
  2. For each time-window, compute your desired metric for volatility. Should preferably be robust to outliers. Median Absolute Deviation is a candidate.
  3. Scale the original data points by the volatility metric to "normalize" / "correct for" volatility.
  4. Do anomaly detection on the new time series. If the feature engineering is strong, this might be as simple as applying a threshold
$\endgroup$
4
  • $\begingroup$ Thank you for your valuable input. What I did following this is to apply local volatility estimates, just as you suggested, to scale the series. In particular, I used the stochvolpackage in R which gave me daily volatility estimates. In the end both the isolation forests and one-class SVM did pretty well. $\endgroup$
    – SimonDude
    Commented Nov 15, 2021 at 13:37
  • $\begingroup$ Nice! Maybe mark this answer as accepted? Or if you wish to provide some more details, make an answer of your own and mark that? $\endgroup$
    – Jon Nordby
    Commented Nov 15, 2021 at 14:30
  • 1
    $\begingroup$ Oh yeah. Sorry, totally forgot. Have a good one $\endgroup$
    – SimonDude
    Commented Nov 16, 2021 at 17:50
  • $\begingroup$ Thanks, have a good week :) $\endgroup$
    – Jon Nordby
    Commented Nov 17, 2021 at 8:23

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.