Method for outlier detection in noisy seasonal time series data?

Question

I have around 1000 times series of around 1000 samples, where each sample is 5 minutes a part.

An example of a time series after performing seasonal decomposition is

As we can see the data is very noisy during the night.

So I am wondering

• What would be a good option for outlier detection in this case? Would any of the following methods make sense

  • Fit a gaussian to residuals and estimate probability for each sample
  • Some threshold for the number of "median absolute deviations" from the median each sample is allowed to have.

• Given a chosen method, what method would make sense to use to dynamically set the threshold depending on how noisy the data is during the night / day.

EDIT Some Sample data

[ 1.55,  1.22,  0.3 , -0.51, -0.17, -0.1 , -2.04, -2.64, -2.31, -0.45,  1.97,  0.71,  0.22, -0.46, -0.48, -0.24, -2.29, -2.06, -1.98, -0.22,  1.84,  0.3 , -0.19, -0.54, -0.37, -0.73, -2.26, -2.16, -1.99, -0.29,  1.36, -0.07, -0.2 , -0.48, -0.87, -0.55, -1.51, -1.75, -2.3 ,  0.12,  0.34, -0.24, -0.28, -0.9 , -0.83,  0.17, -0.62, -1.47, -0.84,  0.78,  1.47,  0.19, -0.1 , -0.31, -0.99, -0.65, -0.51, -1.08, -0.69,  0.2 ,  1.23, -0.49,  0.43, -0.55, -0.73,  0.32, -0.65, -0.72, -0.24,  0.25,  1.5 , -0.08, -0.03, -0.08, -0.38, -0.34,  0.23, -1.07, -0.12,  0.05,  1.3 ,  0.38,  0.02, -0.81, -0.45, -0.54, -0.21, -0.54, -0.18, -0.08,  0.93, -0.69, -0.22, -0.76, -0.31, -0.31, -0.54, -0.54, -0.47,  0.46,  0.54, -0.32,  0.14, -0.32, -0.47, -0.14,  0.12, -0.94, -0.62, -0.24,  0.75,  0.02, -0.62, -0.59, -0.09, -0.62, -0.58, -1.21, -1.1 , -0.58, -0.32, -0.79, -0.35, -0.75, -1.08, -0.52, -0.86, -1.07, -1.78, -0.77,  0.1 ,  0.35, -0.26, -0.56, -0.26, -0.57, -0.66, -1.26, -1.69,  0.58, -0.18, -0.  , -0.36, -0.41, -0.38, -0.85, -0.79, -0.68, -0.99, -0.38, -0.19, -0.5 , -0.23, -0.62,  0.04, -0.47,  0.3 , -1.26, -0.5 ,  0.51, -0.31, -0.15, -0.23, -1.14, -0.3 , -0.33, -0.23, -0.76, -0.9 ,  0.14, -0.05, -0.09,  0.22, -0.19, -0.27, -0.29, -0.58, -1.27, -1.16,  0.07, -0.36, -0.23, -0.22, -0.02, -0.57, -0.9 , -0.08, -0.95, -0.52,  0.63, -0.11,  0.17, -0.49,  0.83,  0.18,  0.14,  0.58,  0.63,  0.94,  1.75,  0.72,  1.19,  0.51,  0.58, -0.43, -1.05, -1.55, -2.91, -2.72, -3.18, -3.39, -2.45,  0.07,  0.02, -1.82, -3.78, -2.91, -3.49, -3.24, -2.55, -0.67,  0.83,  1.87,  2.77,  0.34,  0.17,  1.46,  0.96,  1.55,  1.33, -0.6 ,  0.52,  2.44,  3.07,  0.31,  0.24,  1.23,  0.92,  1.43,  1.15, -0.73,  0.7 ,  2.05,  2.26,  0.53, -0.1 ,  1.01,  0.41,  1.4 ,  1.24, -0.68,  0.74,  2.07,  1.56,  1.09, -0.32,  1.17,  0.55,  1.7 ,  1.06, -0.49,  0.64,  3.1 ,  1.55,  0.88,  0.06,  0.89,  0.45,  1.48,  0.88, -0.22,  0.83,  2.43,  1.7 ,  0.58, -0.16,  0.93,  0.21,  1.04,  0.41, -0.27,  0.94,  1.73,  1.26, -0.51,  0.22,  0.92,  0.34,  0.52, -0.43, -0.3 ,  1.34,  1.53,  1.05,  0.84,  0.87,  1.88,  0.42,  0.57, -0.78, -0.51,  1.26,  1.11,  0.92,  1.3 ,  0.11,  1.71,  0.57,  0.27,  0.17, -0.62,  1.19, -0.19,  1.4 ,  1.03,  0.58,  1.27,  0.65, -0.13,  0.26,  0.76,  0.74,  0.28,  0.82,  0.57,  0.27,  1.12, -0.36,  0.16, -0.6 , -0.34, -0.16,  0.38,  0.35, -0.76,  0.09,  0.59, -0.64, -0.4 , -0.43,  0.63,  0.11,  0.84,  0.38, -0.04,  0.85,  0.47, -0.56, -0.16,  0.28,  0.84, -0.08,  0.32, -0.06, -0.08,  0.6 ,  0.01, -0.69, -0.25, -0.35,  0.45, -0.29,  0.37,  0.15, -0.4 ,  0.29,  0.21, -0.09, -0.46, -0.4 , -0.34,  0.43,  1.2 ,  0.13, -0.36, -0.3 , -0.2 , -0.46,  0.31,  0.28, -0.11,  0.01, -0.22, -0.4 , -0.6 ,  0.37, -0.78, -0.33,  0.38,  0.32, -0.24, -0.13, -0.45, -0.09, -0.48, -0.34, -0.91, -0.1 , -0.05,  0.13,  0.31,  0.04,  0.33,  0.38,  0.02,  0.11, -0.35, -0.2 , -0.87,  0.12, -0.12, -0.12,  0.49,  0.53, -0.02, -0.25, -0.15,  0.2 , -0.51, -0.42,  0.07,  0.25,  0.22,  0.18, -0.45,  0.95,  1.95, -0.64,  0.04,  0.46,  0.24,  0.08, -0.09,  0.08, -0.15,  0.34,  1.22,  0.17,  0.03,  0.21, -0.29,  0.43, -0.38,  0.57, -0.35,  1.24,  0.49, -1.05, -0.06,  0.08,  0.24,  0.66,  0.36,  0.2 , -0.38,  0.09,  0.08, -0.09, -0.61,  0.39,  0.11,  0.39, -0.3 , -0.08,  0.12,  0.84,  0.22,  0.03,  0.1 ,  0.03, -0.22, -0.29,  0.09,  0.38, -0.04,  0.51, -0.51, -0.36,  0.06,  0.56,  0.36, -0.86, -0.02, -0.85, -0.42, -0.47, -0.79, -0.73, -0.74, -0.07, -0.73, -0.19, -0.26,  0.57,  0.51,  0.46, -0.2 ,  1.57,  0.93,  0.59, -1.41, -1.45,  1.19,  3.97,  2.89,  0.89,  0.32,  1.15,  0.39, -0.95, -0.91, -1.77,  1.46,  2.67,  0.97, -0.84, -1.13, -1.14, -1.66, -2.38, -1.09, -2.1 ,  0.97,  2.14,  0.77, -0.7 , -1.46, -1.22, -2.03, -2.36, -0.39, -1.29,  1.1 ,  2.21,  0.59, -0.45, -1.22, -1.36, -2.45, -1.83, -0.15, -0.43,  1.04,  2.8 ,  0.5 , -0.56, -1.41, -1.53, -2.7 , -1.07, -0.79, -0.36,  1.14,  2.43,  0.41, -0.83, -1.12, -1.61, -2.87, -0.76, -0.87, -0.36,  1.42,  2.39, -0.2 , -0.32, -0.96, -1.85, -2.49, -0.85, -0.4 , -0.07,  1.61,  2.33, -0.5 , -0.64, -1.28, -2.18, -1.89, -0.93, -0.41,  0.24,  1.84,  2.83,  0.05, -0.34, -1.96, -2.28, -1.4 , -0.66,  0.24,  0.42,  1.88,  2.4 ,  0.55, -0.4 , -1.67, -1.56, -0.9 , -0.49,  0.75,  0.15,  2.02,  1.46,  0.12, -0.73, -1.46, -1.63, -1.1 , -0.1 ,  0.87, -0.37,  1.83,  0.97,  1.02,  0.04, -0.38, -0.65, -0.44,  0.06,  0.6 , -0.22,  1.38,  0.62,  0.37, -0.55, -0.76, -0.72, -0.4 ,  0.05,  1.1 ,  0.37,  1.06,  0.59,  0.08, -0.31, -0.57, -0.34, -1.21, -0.19,  0.48, -0.04,  1.12,  0.29,  0.15, -0.05, -0.8 , -0.52, -0.73,  0.  ,  0.48, -0.01,  0.11,  0.4 , -0.93, -0.55, -1.25, -0.67, -0.23, -0.04,  0.22,  0.48,  0.92,  0.7 , -0.12,  0.48, -0.89, -0.44,  0.03,  0.39,  0.65,  0.19,  0.94, -0.28,  0.29,  0.19, -0.96, -0.45, -0.18,  0.06,  0.81, -0.14,  0.15,  1.41,  0.53,  0.19, -0.44, -0.17, -0.16, -0.24, -0.68,  0.08,  0.73,  0.14,  0.31,  0.34,  0.52,  0.02,  0.21,  0.26, -0.  , -0.44,  0.96,  0.67,  0.64,  0.24,  0.95,  0.08,  0.23,  0.31,  0.03,  0.39,  1.1 ,  0.31, -0.26,  0.06,  0.13, -0.45,  0.12,  0.32,  0.47,  0.77,  0.94,  0.35, -0.24,  0.21,  0.16,  0.29,  0.52,  0.19,  0.34, -0.1 ,  0.05,  0.02,  0.01,  0.54,  0.37,  0.08, -0.  ,  0.48, -0.06,  0.13,  0.61,  0.67,  0.83, -0.05,  0.66, -0.3 , -0.33, -0.2 ,  0.57,  0.36,  0.45,  0.42,  0.94, -0.1 ,  0.26,  0.2 ,  0.44,  0.31,  0.48,  0.52,  0.13,  0.44,  1.03, -0.27,  0.05, -0.73,  0.13,  0.04, -0.17,  0.71, -0.16, -0.16, -0.15, -1.02,  0.02, -1.12,  0.22, -0.39,  0.69,  0.49,  1.04,  2.45,  2.91,  1.61,  2.46,  1.86,  1.34,  1.43,  0.62, -0.2 ,  0.02,  2.6 ,  2.92,  1.4 ,  0.28,  0.12, -0.96, -1.  , -1.8 , -2.84, -2.43, -0.13, -0.42, -0.19,  2.26,  1.86, -1.36, -0.97, -1.29, -2.39, -2.  , -0.22,  0.03,  0.07,  2.77,  1.66, -1.66, -0.97, -1.63, -2.11, -1.6 ,  0.04, -0.18,  0.12,  3.13,  1.08, -1.92, -1.12, -2.13, -2.48, -1.67,  0.01, -0.29,  0.47,  3.18,  0.43, -2.31, -1.19, -2.02, -2.49, -1.31,  0.38, -0.37,  0.73,  3.09, -0.07, -1.57, -1.34, -2.  , -2.22, -0.72,  0.11, -0.08,  1.44,  2.76, -0.09, -1.33, -1.19, -1.1 , -2.56, -0.42,  0.31, -0.79,  1.39,  1.89,  0.1 , -0.95, -1.2 , -0.65, -1.05,  0.38,  0.38, -0.58,  2.36,  1.69, -0.15, -0.88, -1.11, -0.89, -0.46, -0.29,  0.05, -0.44,  1.09,  1.71, -0.16, -0.19, -0.83, -0.79,  0.12,  0.59,  0.36,  0.23,  1.44,  0.54, -0.15, -0.28,  0.1 , -0.89,  0.52,  0.16,  0.2 , -0.11,  1.49,  1.06,  1.  , -0.15, -0.31, -0.  ,  0.76, -0.13,  0.41, -0.31,  0.96, -0.13,  0.15, -0.96, -0.1 , -0.51, -0.36,  0.14,  0.66, -0.5 ,  0.55, -0.06,  0.82, -0.07, -0.21, -0.39, -0.17,  0.08,  0.49, -0.44,  0.95,  0.31,  0.36, -0.47,  0.19,  0.06,  0.38,  0.84,  0.59,  0.4 ,  0.69,  0.55,  0.42, -0.96, -0.07, -0.35,  0.15,  0.5 ,  0.06, -0.35,  0.84,  0.29,  0.36, -0.12,  0.52,  0.2 ,  0.46,  0.96, -0.31,  0.04,  0.46,  0.28,  0.39,  0.11,  0.37,  0.21, -0.13,  0.99,  0.15, -0.27,  0.01,  0.48,  0.78,  0.44,  0.16, -0.18,  0.96,  1.14,  0.44,  0.67,  0.65,  0.26,  0.62,  0.6 ,  0.43, -0.09,  0.65,  1.3 ,  0.33, -0.54, -0.02, -0.04]

Answer 1

Just by eye, it seems like one of the important characteristics of your time series is the distinct night and day behaviour. So what would be an outlier during the day won’t necessarily be an outlier at night.

In this case, one could try:

• Fit a HMM to learn the two state behaviour and detect outliers by examining $P(O|X)$ (probability of observation given hidden state).
• Calculating a moving Z-score to detect how much of an outlier the observation is to recent values. (Or Z-score calculated relative to values from a similar time on the previous day).