1
$\begingroup$

I have around 1000 times series of around 1000 samples, where each sample is 5 minutes a part.

An example of a time series after performing seasonal decomposition is

enter image description here

As we can see the data is very noisy during the night.

So I am wondering

  • What would be a good option for outlier detection in this case? Would any of the following methods make sense

    • Fit a gaussian to residuals and estimate probability for each sample
    • Some threshold for the number of "median absolute deviations" from the median each sample is allowed to have.
  • Given a chosen method, what method would make sense to use to dynamically set the threshold depending on how noisy the data is during the night / day.

EDIT Some Sample data

[ 1.55,  1.22,  0.3 , -0.51, -0.17, -0.1 , -2.04, -2.64, -2.31, -0.45,  1.97,  0.71,  0.22, -0.46, -0.48, -0.24, -2.29, -2.06, -1.98, -0.22,  1.84,  0.3 , -0.19, -0.54, -0.37, -0.73, -2.26, -2.16, -1.99, -0.29,  1.36, -0.07, -0.2 , -0.48, -0.87, -0.55, -1.51, -1.75, -2.3 ,  0.12,  0.34, -0.24, -0.28, -0.9 , -0.83,  0.17, -0.62, -1.47, -0.84,  0.78,  1.47,  0.19, -0.1 , -0.31, -0.99, -0.65, -0.51, -1.08, -0.69,  0.2 ,  1.23, -0.49,  0.43, -0.55, -0.73,  0.32, -0.65, -0.72, -0.24,  0.25,  1.5 , -0.08, -0.03, -0.08, -0.38, -0.34,  0.23, -1.07, -0.12,  0.05,  1.3 ,  0.38,  0.02, -0.81, -0.45, -0.54, -0.21, -0.54, -0.18, -0.08,  0.93, -0.69, -0.22, -0.76, -0.31, -0.31, -0.54, -0.54, -0.47,  0.46,  0.54, -0.32,  0.14, -0.32, -0.47, -0.14,  0.12, -0.94, -0.62, -0.24,  0.75,  0.02, -0.62, -0.59, -0.09, -0.62, -0.58, -1.21, -1.1 , -0.58, -0.32, -0.79, -0.35, -0.75, -1.08, -0.52, -0.86, -1.07, -1.78, -0.77,  0.1 ,  0.35, -0.26, -0.56, -0.26, -0.57, -0.66, -1.26, -1.69,  0.58, -0.18, -0.  , -0.36, -0.41, -0.38, -0.85, -0.79, -0.68, -0.99, -0.38, -0.19, -0.5 , -0.23, -0.62,  0.04, -0.47,  0.3 , -1.26, -0.5 ,  0.51, -0.31, -0.15, -0.23, -1.14, -0.3 , -0.33, -0.23, -0.76, -0.9 ,  0.14, -0.05, -0.09,  0.22, -0.19, -0.27, -0.29, -0.58, -1.27, -1.16,  0.07, -0.36, -0.23, -0.22, -0.02, -0.57, -0.9 , -0.08, -0.95, -0.52,  0.63, -0.11,  0.17, -0.49,  0.83,  0.18,  0.14,  0.58,  0.63,  0.94,  1.75,  0.72,  1.19,  0.51,  0.58, -0.43, -1.05, -1.55, -2.91, -2.72, -3.18, -3.39, -2.45,  0.07,  0.02, -1.82, -3.78, -2.91, -3.49, -3.24, -2.55, -0.67,  0.83,  1.87,  2.77,  0.34,  0.17,  1.46,  0.96,  1.55,  1.33, -0.6 ,  0.52,  2.44,  3.07,  0.31,  0.24,  1.23,  0.92,  1.43,  1.15, -0.73,  0.7 ,  2.05,  2.26,  0.53, -0.1 ,  1.01,  0.41,  1.4 ,  1.24, -0.68,  0.74,  2.07,  1.56,  1.09, -0.32,  1.17,  0.55,  1.7 ,  1.06, -0.49,  0.64,  3.1 ,  1.55,  0.88,  0.06,  0.89,  0.45,  1.48,  0.88, -0.22,  0.83,  2.43,  1.7 ,  0.58, -0.16,  0.93,  0.21,  1.04,  0.41, -0.27,  0.94,  1.73,  1.26, -0.51,  0.22,  0.92,  0.34,  0.52, -0.43, -0.3 ,  1.34,  1.53,  1.05,  0.84,  0.87,  1.88,  0.42,  0.57, -0.78, -0.51,  1.26,  1.11,  0.92,  1.3 ,  0.11,  1.71,  0.57,  0.27,  0.17, -0.62,  1.19, -0.19,  1.4 ,  1.03,  0.58,  1.27,  0.65, -0.13,  0.26,  0.76,  0.74,  0.28,  0.82,  0.57,  0.27,  1.12, -0.36,  0.16, -0.6 , -0.34, -0.16,  0.38,  0.35, -0.76,  0.09,  0.59, -0.64, -0.4 , -0.43,  0.63,  0.11,  0.84,  0.38, -0.04,  0.85,  0.47, -0.56, -0.16,  0.28,  0.84, -0.08,  0.32, -0.06, -0.08,  0.6 ,  0.01, -0.69, -0.25, -0.35,  0.45, -0.29,  0.37,  0.15, -0.4 ,  0.29,  0.21, -0.09, -0.46, -0.4 , -0.34,  0.43,  1.2 ,  0.13, -0.36, -0.3 , -0.2 , -0.46,  0.31,  0.28, -0.11,  0.01, -0.22, -0.4 , -0.6 ,  0.37, -0.78, -0.33,  0.38,  0.32, -0.24, -0.13, -0.45, -0.09, -0.48, -0.34, -0.91, -0.1 , -0.05,  0.13,  0.31,  0.04,  0.33,  0.38,  0.02,  0.11, -0.35, -0.2 , -0.87,  0.12, -0.12, -0.12,  0.49,  0.53, -0.02, -0.25, -0.15,  0.2 , -0.51, -0.42,  0.07,  0.25,  0.22,  0.18, -0.45,  0.95,  1.95, -0.64,  0.04,  0.46,  0.24,  0.08, -0.09,  0.08, -0.15,  0.34,  1.22,  0.17,  0.03,  0.21, -0.29,  0.43, -0.38,  0.57, -0.35,  1.24,  0.49, -1.05, -0.06,  0.08,  0.24,  0.66,  0.36,  0.2 , -0.38,  0.09,  0.08, -0.09, -0.61,  0.39,  0.11,  0.39, -0.3 , -0.08,  0.12,  0.84,  0.22,  0.03,  0.1 ,  0.03, -0.22, -0.29,  0.09,  0.38, -0.04,  0.51, -0.51, -0.36,  0.06,  0.56,  0.36, -0.86, -0.02, -0.85, -0.42, -0.47, -0.79, -0.73, -0.74, -0.07, -0.73, -0.19, -0.26,  0.57,  0.51,  0.46, -0.2 ,  1.57,  0.93,  0.59, -1.41, -1.45,  1.19,  3.97,  2.89,  0.89,  0.32,  1.15,  0.39, -0.95, -0.91, -1.77,  1.46,  2.67,  0.97, -0.84, -1.13, -1.14, -1.66, -2.38, -1.09, -2.1 ,  0.97,  2.14,  0.77, -0.7 , -1.46, -1.22, -2.03, -2.36, -0.39, -1.29,  1.1 ,  2.21,  0.59, -0.45, -1.22, -1.36, -2.45, -1.83, -0.15, -0.43,  1.04,  2.8 ,  0.5 , -0.56, -1.41, -1.53, -2.7 , -1.07, -0.79, -0.36,  1.14,  2.43,  0.41, -0.83, -1.12, -1.61, -2.87, -0.76, -0.87, -0.36,  1.42,  2.39, -0.2 , -0.32, -0.96, -1.85, -2.49, -0.85, -0.4 , -0.07,  1.61,  2.33, -0.5 , -0.64, -1.28, -2.18, -1.89, -0.93, -0.41,  0.24,  1.84,  2.83,  0.05, -0.34, -1.96, -2.28, -1.4 , -0.66,  0.24,  0.42,  1.88,  2.4 ,  0.55, -0.4 , -1.67, -1.56, -0.9 , -0.49,  0.75,  0.15,  2.02,  1.46,  0.12, -0.73, -1.46, -1.63, -1.1 , -0.1 ,  0.87, -0.37,  1.83,  0.97,  1.02,  0.04, -0.38, -0.65, -0.44,  0.06,  0.6 , -0.22,  1.38,  0.62,  0.37, -0.55, -0.76, -0.72, -0.4 ,  0.05,  1.1 ,  0.37,  1.06,  0.59,  0.08, -0.31, -0.57, -0.34, -1.21, -0.19,  0.48, -0.04,  1.12,  0.29,  0.15, -0.05, -0.8 , -0.52, -0.73,  0.  ,  0.48, -0.01,  0.11,  0.4 , -0.93, -0.55, -1.25, -0.67, -0.23, -0.04,  0.22,  0.48,  0.92,  0.7 , -0.12,  0.48, -0.89, -0.44,  0.03,  0.39,  0.65,  0.19,  0.94, -0.28,  0.29,  0.19, -0.96, -0.45, -0.18,  0.06,  0.81, -0.14,  0.15,  1.41,  0.53,  0.19, -0.44, -0.17, -0.16, -0.24, -0.68,  0.08,  0.73,  0.14,  0.31,  0.34,  0.52,  0.02,  0.21,  0.26, -0.  , -0.44,  0.96,  0.67,  0.64,  0.24,  0.95,  0.08,  0.23,  0.31,  0.03,  0.39,  1.1 ,  0.31, -0.26,  0.06,  0.13, -0.45,  0.12,  0.32,  0.47,  0.77,  0.94,  0.35, -0.24,  0.21,  0.16,  0.29,  0.52,  0.19,  0.34, -0.1 ,  0.05,  0.02,  0.01,  0.54,  0.37,  0.08, -0.  ,  0.48, -0.06,  0.13,  0.61,  0.67,  0.83, -0.05,  0.66, -0.3 , -0.33, -0.2 ,  0.57,  0.36,  0.45,  0.42,  0.94, -0.1 ,  0.26,  0.2 ,  0.44,  0.31,  0.48,  0.52,  0.13,  0.44,  1.03, -0.27,  0.05, -0.73,  0.13,  0.04, -0.17,  0.71, -0.16, -0.16, -0.15, -1.02,  0.02, -1.12,  0.22, -0.39,  0.69,  0.49,  1.04,  2.45,  2.91,  1.61,  2.46,  1.86,  1.34,  1.43,  0.62, -0.2 ,  0.02,  2.6 ,  2.92,  1.4 ,  0.28,  0.12, -0.96, -1.  , -1.8 , -2.84, -2.43, -0.13, -0.42, -0.19,  2.26,  1.86, -1.36, -0.97, -1.29, -2.39, -2.  , -0.22,  0.03,  0.07,  2.77,  1.66, -1.66, -0.97, -1.63, -2.11, -1.6 ,  0.04, -0.18,  0.12,  3.13,  1.08, -1.92, -1.12, -2.13, -2.48, -1.67,  0.01, -0.29,  0.47,  3.18,  0.43, -2.31, -1.19, -2.02, -2.49, -1.31,  0.38, -0.37,  0.73,  3.09, -0.07, -1.57, -1.34, -2.  , -2.22, -0.72,  0.11, -0.08,  1.44,  2.76, -0.09, -1.33, -1.19, -1.1 , -2.56, -0.42,  0.31, -0.79,  1.39,  1.89,  0.1 , -0.95, -1.2 , -0.65, -1.05,  0.38,  0.38, -0.58,  2.36,  1.69, -0.15, -0.88, -1.11, -0.89, -0.46, -0.29,  0.05, -0.44,  1.09,  1.71, -0.16, -0.19, -0.83, -0.79,  0.12,  0.59,  0.36,  0.23,  1.44,  0.54, -0.15, -0.28,  0.1 , -0.89,  0.52,  0.16,  0.2 , -0.11,  1.49,  1.06,  1.  , -0.15, -0.31, -0.  ,  0.76, -0.13,  0.41, -0.31,  0.96, -0.13,  0.15, -0.96, -0.1 , -0.51, -0.36,  0.14,  0.66, -0.5 ,  0.55, -0.06,  0.82, -0.07, -0.21, -0.39, -0.17,  0.08,  0.49, -0.44,  0.95,  0.31,  0.36, -0.47,  0.19,  0.06,  0.38,  0.84,  0.59,  0.4 ,  0.69,  0.55,  0.42, -0.96, -0.07, -0.35,  0.15,  0.5 ,  0.06, -0.35,  0.84,  0.29,  0.36, -0.12,  0.52,  0.2 ,  0.46,  0.96, -0.31,  0.04,  0.46,  0.28,  0.39,  0.11,  0.37,  0.21, -0.13,  0.99,  0.15, -0.27,  0.01,  0.48,  0.78,  0.44,  0.16, -0.18,  0.96,  1.14,  0.44,  0.67,  0.65,  0.26,  0.62,  0.6 ,  0.43, -0.09,  0.65,  1.3 ,  0.33, -0.54, -0.02, -0.04]
$\endgroup$

1 Answer 1

1
$\begingroup$

Just by eye, it seems like one of the important characteristics of your time series is the distinct night and day behaviour. So what would be an outlier during the day won’t necessarily be an outlier at night.

In this case, one could try:

  • Fit a HMM to learn the two state behaviour and detect outliers by examining $P(O|X)$ (probability of observation given hidden state).
  • Calculating a moving Z-score to detect how much of an outlier the observation is to recent values. (Or Z-score calculated relative to values from a similar time on the previous day).
$\endgroup$
2
  • $\begingroup$ Thanks for the answer! For the second approach, do you think one would have a moving Z-score that has no window overlap between day and night? $\endgroup$
    – kspr
    Oct 5, 2021 at 6:41
  • $\begingroup$ Yeah that's a good question. I'm not sure what makes the most sense. The two traps are i) unintentionally classifying the start of the night period as an outlier and ii) not catching a true outlier at the beginning of night/day. You could also maybe try forecasting the next point and then classify an outlier based on how 'unexpected' it is (different from your prediction). $\endgroup$
    – Adam Kells
    Oct 5, 2021 at 9:15

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.