How to detect abnormality in an otherwise very systematic and regular time-series data for temperature measurement?

Question

I have time-series data, let's say a pandas series, with time (sampling frequency is hourly) as its index and temperature measurement across that time. I want some statistical/time-series principle which can tell whether a time-series is well-behaved or not.

What I mean by well behaved time-series is that, let's say the distribution of temperature for a day is same/almost identical for all 7 or even 30 days of the month. The reason for detecting even a slight deviation is to know whether some sensors that collect temperature are working properly or not. The device, whose temperature sensors are measuring every hour, has the property that it's temperature distribution for the whole day remains almost identical throughout the month.

Answer 1

Maybe start simple. If you are expecting distributions to be identical day to day, test each day's against the baseline (whatever you consider normal): http://www.itl.nist.gov/div898/software/dataplot/refman1/auxillar/ks2samp.htm

If you are looking to anomaly detection intraday, and you have a good model for the distribution, can you just have a probability cut-off for outliers?

Answer 2

Detecting the onset of unusual activity is the subject of outlier detection and nearly about every answer that i have recently made. A model reflecting period to period dependency and/or day-to=day dependency can be developed using Transfer Function/Dynamic Regression while "unusual" innovation can be detected when typical rules fail. If you wish to post your data I would be happy to take a look at it and hopefully other readers would do the same. Following is a very good thread with respect to anomaly (intervention) detection.Detecting Outliers in Time Series (LS/AO/TC) using tsoutliers package in R. How to represent outliers in equation format? . Read all the answers and comments and particularly closely follow the Tsay 1986 article http://www.unc.edu/~jbhill/tsay.pdf

Answer 3

I think the best method for identifying sensor problems from time series data is to test for stationarity rather than outliers or anomalies alone. Outliers are individual data points that lie outside the expected or normal range. Anomolies are patterns of data points that are somehow distinct or "not typical", even though the might be inside the normal or expected range.

In contrast, non-stationary time series is a time series where the generating distribution has changed or is changing over time. In other words, stationarity is concerned with the generating distribution and not with individual data points or groups of data points. As you said, the distributions associated with working sensors stays the same (i.e. is "stationary") over a month.

Here are a few introductory references:

• http://www.maths.bris.ac.uk/~guy/Research/LSTS/TOS.html
• https://quant.stackexchange.com/questions/2372/how-to-check-if-a-timeseries-is-stationary
• http://www.cas.usf.edu/~cconnor/geolsoc/html/chapter11.pdf

The problem with outlier detection as a method is that there might be many causes of outliers not relate to faults in sensors. Same for anomalies. It might be true that some changes in stationarity might also be accompanied by either outliers or anomalies, but that is not necessarily the case. In contrast, changes in stationarity will almost always be related to faults or failures in sensors and related processes of data capture and transmission.

The downside of stationarity tests is that it is hard to detect changes in stationarity quickly in real time, with high reliability (i.e. minimum of false positives). If you might combine several methods to get "early warning signals" of possible sensor problems, and then confirm them later (hours or days) after more data comes in.

Answer 4

1. Let us assume that last K days you have measurement which you can trust and are OK.
2. You are now interested to see if day K+1 distribution is the same as in the previous K days. To check that you can do a Two-sample Kolmogorov-Smirnov test

Example (R):

library(data.table)

set.seed(34976742)

# daily pattern
DT <- data.table(h=1:24, base = rlogis(24, 20, 2))
# number of days in history
K <- 20

# simulated historical data
historical.DT <- DT[, list(day = 1:K, t = rnorm(K, base, .5)), by = h]
# simulated test day data
new.DT <- historical.DT[, list(day = K+1, t = rnorm(1, mean(t), 1)), by = h]

# Two-sample Kolmogorov-Smirnov test
ks.test(historical.DT[, t], new.DT[,t])

Note that historical.DT[, t] is a vector of measurement ordered first by hour, then by day, while new.DT[,t] is ordered by hour.