2
$\begingroup$

I have time-series data, let's say a pandas series, with time (sampling frequency is hourly) as its index and temperature measurement across that time. I want some statistical/time-series principle which can tell whether a time-series is well-behaved or not.

What I mean by well behaved time-series is that, let's say the distribution of temperature for a day is same/almost identical for all 7 or even 30 days of the month. The reason for detecting even a slight deviation is to know whether some sensors that collect temperature are working properly or not. The device, whose temperature sensors are measuring every hour, has the property that it's temperature distribution for the whole day remains almost identical throughout the month.

$\endgroup$
  • $\begingroup$ distribution remains same for whole month? Is this measuring temperatures of a fridge or sthg like that. $\endgroup$ – adam Jul 22 '15 at 16:06
  • $\begingroup$ well, it's just a hypothetical scenario. I can't really talk about why i need this but i think the description specifies my problem very closely, so i need some help. $\endgroup$ – lovekesh Jul 22 '15 at 18:24
  • $\begingroup$ specification of the problem often requires data. If you can't post your data... then transform your data and post the transformed data. This might help to draw out precisely what you need to do OR can't do . $\endgroup$ – IrishStat Jul 23 '15 at 21:09
  • $\begingroup$ @IrishStat I will post a dummy data very soon. Thanks again. $\endgroup$ – lovekesh Jul 24 '15 at 5:55
  • $\begingroup$ This question is so vague and general that no one will be able to tell whether any of the (extremely different) proposed answers is any good for your situation. All those that have appeared so far implicitly adopt relatively strong (but differing) assumptions about your data and about the kind of "abnormality" you are looking for. If you could be more specific about those two things you would likely get better guidance. $\endgroup$ – whuber Jul 25 '15 at 13:58
2
+25
$\begingroup$

Maybe start simple. If you are expecting distributions to be identical day to day, test each day's against the baseline (whatever you consider normal): http://www.itl.nist.gov/div898/software/dataplot/refman1/auxillar/ks2samp.htm

If you are looking to anomaly detection intraday, and you have a good model for the distribution, can you just have a probability cut-off for outliers?

$\endgroup$
  • $\begingroup$ What you are suggesting is an outdated out-of-model test which is flawed by the effect of the unincorporated outliers/level shifts/seasonal pulse/local time trends while Intervention Detection is a "probability cut-off" based upon a within-model test that can lead to the emorical identification of not only pulses BUT level shifts/seasonal pulses and local time trends. $\endgroup$ – IrishStat Jul 23 '15 at 17:15
  • $\begingroup$ Agreed, but the way the problem was presented makes it sound like seasonality and level shifts are not an issue :) $\endgroup$ – gbasin Jul 23 '15 at 17:19
  • $\begingroup$ @gbasin Yes, you are right. seasonality and level shifts is not an issue. $\endgroup$ – lovekesh Jul 24 '15 at 5:54
1
$\begingroup$

Detecting the onset of unusual activity is the subject of outlier detection and nearly about every answer that i have recently made. A model reflecting period to period dependency and/or day-to=day dependency can be developed using Transfer Function/Dynamic Regression while "unusual" innovation can be detected when typical rules fail. If you wish to post your data I would be happy to take a look at it and hopefully other readers would do the same. Following is a very good thread with respect to anomaly (intervention) detection.Detecting Outliers in Time Series (LS/AO/TC) using tsoutliers package in R. How to represent outliers in equation format? . Read all the answers and comments and particularly closely follow the Tsay 1986 article http://www.unc.edu/~jbhill/tsay.pdf

$\endgroup$
  • $\begingroup$ Question is related to outlier detection in time series, your link does not provide any information on this $\endgroup$ – adam Jul 22 '15 at 16:04
  • $\begingroup$ @adam which link are you referring to ? All my links refer to time series as that is the only subject/topic I know . $\endgroup$ – IrishStat Jul 22 '15 at 16:09
  • $\begingroup$ Both links go to the same answer of yours, which is quite general and does not clearly address outlier detection at all. Since you have posted often, and sometimes in detail, about outlier detection in time series, I'm confident you could find a better reference than that! $\endgroup$ – whuber Jul 22 '15 at 16:19
  • 1
    $\begingroup$ @whuber sorry about that ... I have expanded my answer .... $\endgroup$ – IrishStat Jul 22 '15 at 18:36
  • $\begingroup$ Have you searched the literature on robust control charts? You could start here $\endgroup$ – user603 Jul 22 '15 at 21:25
1
$\begingroup$

I think the best method for identifying sensor problems from time series data is to test for stationarity rather than outliers or anomalies alone. Outliers are individual data points that lie outside the expected or normal range. Anomolies are patterns of data points that are somehow distinct or "not typical", even though the might be inside the normal or expected range.

In contrast, non-stationary time series is a time series where the generating distribution has changed or is changing over time. In other words, stationarity is concerned with the generating distribution and not with individual data points or groups of data points. As you said, the distributions associated with working sensors stays the same (i.e. is "stationary") over a month.

Here are a few introductory references:

The problem with outlier detection as a method is that there might be many causes of outliers not relate to faults in sensors. Same for anomalies. It might be true that some changes in stationarity might also be accompanied by either outliers or anomalies, but that is not necessarily the case. In contrast, changes in stationarity will almost always be related to faults or failures in sensors and related processes of data capture and transmission.

The downside of stationarity tests is that it is hard to detect changes in stationarity quickly in real time, with high reliability (i.e. minimum of false positives). If you might combine several methods to get "early warning signals" of possible sensor problems, and then confirm them later (hours or days) after more data comes in.

$\endgroup$
  • $\begingroup$ ARIMA models are non-stationary and easily extend to Intervention Detection $\endgroup$ – IrishStat Jul 24 '15 at 22:53
  • $\begingroup$ @IrishStat Yes, ARIMA models can be used to detect non-stationarity. I presume it has strengths and limitations, but I don't have experience with it so I won't comment further. $\endgroup$ – MrMeritology Jul 24 '15 at 23:08
1
$\begingroup$
  1. Let us assume that last K days you have measurement which you can trust and are OK.
  2. You are now interested to see if day K+1 distribution is the same as in the previous K days. To check that you can do a Two-sample Kolmogorov-Smirnov test

Example (R):

library(data.table)

set.seed(34976742)

# daily pattern
DT <- data.table(h=1:24, base = rlogis(24, 20, 2))
# number of days in history
K <- 20

# simulated historical data
historical.DT <- DT[, list(day = 1:K, t = rnorm(K, base, .5)), by = h]
# simulated test day data
new.DT <- historical.DT[, list(day = K+1, t = rnorm(1, mean(t), 1)), by = h]

# Two-sample Kolmogorov-Smirnov test
ks.test(historical.DT[, t], new.DT[,t])

Note that historical.DT[, t] is a vector of measurement ordered first by hour, then by day, while new.DT[,t] is ordered by hour.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.