Method to remove bad values in time series (bad values known to take on a particular value)

Question

This sounds easy, but I don't know of a good statistical method for it.

I have a time series that has (good) data points that range from ~3.5 to 30. The data are collected by an automated sensor. However, there are flawed measurements in the time series-- the sensor will sometimes read values that are typically at exactly 4.7. Example (in R):

Data <- c(10.7,  4.7, 10.7,  4.7, 11.5,  4.7,  8.4,  4.7, 11.0,  4.7, 10.8,  4.7, 12.0,  10.3,  4.7, 10.4, 10.9,  4.7, 10.8,  4.7, 11.1,  4.7,  8.8, 4.7,  6.4,  4.7,  9.1,  4.7,  9.1,  4.7,  9.7,  4.7,  7.8,  4.7,  7.2,  4.7,  6.2,  4.7,  8.7,  4.7,  9.0,  4.7,  9.6,  4.7,  7.9,  4.7,  9.2,4.7,  8.1,  4.7,  7.4,  4.7,  7.6,  9.3,  4.7,  9.2,  4.7,  8.8,  4.7,  9.3,  4.7,  9.2,  4.7,  7.0,  4.7,  9.2,  4.7,  8.5,  4.7,  6.2,4.7,  7.1,  4.7,   7.4,  4.7,  8.0,  4.7,  7.3,  4.7,  6.6,  4.7,  6.9,  4.7,  7.2,  7.9,  4.7,  9.0,  4.7,  8.7,  4.7,  8.2,  4.7,  5.1,4.7,  5.6,  4.7,  7.0,  4.7,  9.4,  4.7,  7.6,  4.7,  8.6,  4.7,  9.3,  4.7,  9.7,  4.7, 10.4, 4.7,  10.6,  4.7, 10.9,  4.7, 10.2,  4.7, 10.0,4.7,  8.3,  10.0, 8.7,  4.7, 10.2,  9.7,  4.7, 10.2,  4.7, 10.6,  4.7, 10.5,  4.7,  9.7,  4.7, 10.6,  4.7, 10.6,  4.7, 11.6,4.7, 11.4,  4.7, 10.2,  4.7, 10.3,  4.7, 10.0,  4.7,  9.3,  4.7,  9.1,  4.7, 10.2,  4.7,  8.5,  4.7,  7.2,  4.7, 10.4,  4.7, 10.4, 10.0,4.7,  9.9,  4.7, 11.2, 9.5,  4.7, 11.5,  4.7, 11.5,  4.7, 11.1,  4.7, 11.4,  4.7, 12.0,  4.7, 11.4,  4.7, 11.2,  4.7,  9.9,  4.7, 11.6, 4.7, 14.4,  4.7, 11.5,  4.7, 11.1,  4.7, 11.3,  4.7, 10.9,  4.7, 11.1,  4.7, 11.1,  4.7, 10.9, 11.4,  4.7, 12.6,  4.7, 11.0,  11.3)

plot(1:length(test), test, type="o")

As you can see, almost every other point is 4.7. I don't want to simply use the rule of "throw out every value that is equal to 4.7" for two reasons: 1) sometimes there are real values that approach 4.7, which can become apparent as you see the slope of the time series change and enter the 4.7 region; 2) the "bad" values aren't always exactly 4.7 (they almost always are, but there is no guarantee).

This seems like a fairly simple problem, and I'm hoping that someone has heard of a statistical approach that would be suitiable for this problem. The desired outcome would be a time series where the "bad" values were statistically flagged (e.g., with a probability of being an error).

Suggestions? Thanks!

EDIT: I'm still mulling over the first two responses. My original inclination was to define two states of the system: an erroneous state and a measured state. The measured state would reflect a modeled process (e.g., AR). The erroneous state would reflect my knowledge that bad values tend to be near 4.7. Both states could ~N(), with appropriate means and sd's. At each time step I could calculate the likelihood of either state, and define a threshold ratio for declaring an obs erroneous. This is similar to the 2 answers. For this approach, I'm stuck at finding a way to let the model for the measured state make estimates w/o being influenced by bad values. However, as stated above, I'm still considering the other two approaches.

Answer 1

I would guess that you could take every value that was 4.7 or so, and use

• Various interpolation rules to replace it. These rules could start with linear, cubic, cubic spline or piecewise cubic Hermite interpolation. On the last, see http://en.wikipedia.org/wiki/Cubic_Hermite_spline

• Any appropriate time series smoothing or forecasting method (e.g. some kind of exponential smoothing) to replace it.

If the 4.7 is genuine, a good method should replace it with about the same value any way (that's optimistic, but I don't know what other attitude to take).

How would you assess which possible method was least bad? Take genuine values not 4.7, replace them with missing, then see which method got closest, using a criterion for closest that matches your data generation process and statistical philosophy.

Answer 2

(I wouldn't call this a simple problem. Certainly my suggested solution isn't simple.)

From your description you have no way to be certain if a particular observation of 4.7 is genuine or not; if the series is near 4.7 you could have any number of 'legitimate' 4.7's mixed with any number of 'bad' ones.

As such, any assessment would probably be best regarded probabilistically.

I'd be inclined toward a Bayesian (indeed, MCMC) approach that has a model for the generation of the 'bad' 4.7-s and (from the model for the rest of the data) is able to infer the lost information on any iteration it decides an observation near 4.7 is bad.

You'd end up with some points near 4.7 being always or almost always regarded as 'bad' (the ones where it's obviously bad) and then it's treated like missing data, and some points near 4.7 that are mostly regarded as 'good', which are occasionally replaced with a value suggested by the other data and your model (which will also turn out to be more or less near 4.7), and some data that's 'in between' (where you might suspect it could be bad but wouldn't be sure), which is often but not always replaced.