0
$\begingroup$

Suppose that I have as input a time series $T = \{ t_1, t_2, ..., t_M \}$ where each point is sampled at a fixed time interval (e.g. every 10 ms).

The problem is that $T$ contains a lot of periods with constant values. For example, the values from timestamp 50 (ms) to timestamp 10000 (ms) are the same. Then the data fluctuate from 10010 (ms) to 10200 (ms), then it remains constant until 15000 (ms). So, the series is generally idle until a change happens.

Seeing that keeping these "idle" periods will not bring any insights to the analysis I want to do afterward (for example, clustering or motif discovery), I think those periods can be eliminated from my data, such that any consecutive data points in $T$ are now different. But I still keep the timestamp of each data point to know their arrival time.

My question is: is it safe, or correct, to do so? After the elimination, $T$ will no longer be an evenly spaced time series. Is this going to affect the analysis?

I want to note that mostly I want to do shape-based analysis on the series, e.g. clustering, motif discovery, anomaly detection, etc. Forecasting is not of my interest.

Many thanks in advance!

$\endgroup$

1 Answer 1

1
$\begingroup$

In general: No, you cannot simply drop these "idle" values, and then resume as if it were an evenly sampled timeseries. For some specific methods it might not matter to drop these values, but in general it does. Consider for example anomaly detection, which you mentioned. A simple way of anomaly detection is to test whether a given value is above a certain quantile of your observations. If you drop your constant values before computing this quantile, it will be very different. It might be that for a specific problem you actually want this different quantile. But in order to interpret the result correctly, you then must treat your new timeseries (the one after dropping the "idle" values) as a specially filtered one, and one not identical to your original one.

For many problems, the "idle" values might actually be very interesting. Let's say you want to know the typical persistence of a value (how long this value stays the same). Here these "idle" values will actually have the highest persistence. (Of course, again, depending on the problem setting, you might not be interested in them, but the results would not be the same when you drop the "idle" values)

One could also argue that dropping the "idle" values, and retaining the timestep and the constant value (e.g. drop 100 consecutive values with value 1, but retain the information that you dropped 100 variables and that they all had value 1) is a form of lossless data-compression, because you can completely recover your timeseries from this. In some specific applications this might be a good idea, but any algorithm that you apply on this compressed timeseries will have to take this into account, and cannot simply proceed as if it were the complete timeseries.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.