# Method to remove bad values in time series (bad values known to take on a particular value)

This sounds easy, but I don't know of a good statistical method for it.

I have a time series that has (good) data points that range from ~3.5 to 30. The data are collected by an automated sensor. However, there are flawed measurements in the time series-- the sensor will sometimes read values that are typically at exactly 4.7. Example (in R):

Data <- c(10.7,  4.7, 10.7,  4.7, 11.5,  4.7,  8.4,  4.7, 11.0,  4.7, 10.8,  4.7, 12.0,  10.3,  4.7, 10.4, 10.9,  4.7, 10.8,  4.7, 11.1,  4.7,  8.8, 4.7,  6.4,  4.7,  9.1,  4.7,  9.1,  4.7,  9.7,  4.7,  7.8,  4.7,  7.2,  4.7,  6.2,  4.7,  8.7,  4.7,  9.0,  4.7,  9.6,  4.7,  7.9,  4.7,  9.2,4.7,  8.1,  4.7,  7.4,  4.7,  7.6,  9.3,  4.7,  9.2,  4.7,  8.8,  4.7,  9.3,  4.7,  9.2,  4.7,  7.0,  4.7,  9.2,  4.7,  8.5,  4.7,  6.2,4.7,  7.1,  4.7,   7.4,  4.7,  8.0,  4.7,  7.3,  4.7,  6.6,  4.7,  6.9,  4.7,  7.2,  7.9,  4.7,  9.0,  4.7,  8.7,  4.7,  8.2,  4.7,  5.1,4.7,  5.6,  4.7,  7.0,  4.7,  9.4,  4.7,  7.6,  4.7,  8.6,  4.7,  9.3,  4.7,  9.7,  4.7, 10.4, 4.7,  10.6,  4.7, 10.9,  4.7, 10.2,  4.7, 10.0,4.7,  8.3,  10.0, 8.7,  4.7, 10.2,  9.7,  4.7, 10.2,  4.7, 10.6,  4.7, 10.5,  4.7,  9.7,  4.7, 10.6,  4.7, 10.6,  4.7, 11.6,4.7, 11.4,  4.7, 10.2,  4.7, 10.3,  4.7, 10.0,  4.7,  9.3,  4.7,  9.1,  4.7, 10.2,  4.7,  8.5,  4.7,  7.2,  4.7, 10.4,  4.7, 10.4, 10.0,4.7,  9.9,  4.7, 11.2, 9.5,  4.7, 11.5,  4.7, 11.5,  4.7, 11.1,  4.7, 11.4,  4.7, 12.0,  4.7, 11.4,  4.7, 11.2,  4.7,  9.9,  4.7, 11.6, 4.7, 14.4,  4.7, 11.5,  4.7, 11.1,  4.7, 11.3,  4.7, 10.9,  4.7, 11.1,  4.7, 11.1,  4.7, 10.9, 11.4,  4.7, 12.6,  4.7, 11.0,  11.3)

plot(1:length(test), test, type="o")


As you can see, almost every other point is 4.7. I don't want to simply use the rule of "throw out every value that is equal to 4.7" for two reasons: 1) sometimes there are real values that approach 4.7, which can become apparent as you see the slope of the time series change and enter the 4.7 region; 2) the "bad" values aren't always exactly 4.7 (they almost always are, but there is no guarantee).

This seems like a fairly simple problem, and I'm hoping that someone has heard of a statistical approach that would be suitiable for this problem. The desired outcome would be a time series where the "bad" values were statistically flagged (e.g., with a probability of being an error).

Suggestions? Thanks!

EDIT: I'm still mulling over the first two responses. My original inclination was to define two states of the system: an erroneous state and a measured state. The measured state would reflect a modeled process (e.g., AR). The erroneous state would reflect my knowledge that bad values tend to be near 4.7. Both states could ~N(), with appropriate means and sd's. At each time step I could calculate the likelihood of either state, and define a threshold ratio for declaring an obs erroneous. This is similar to the 2 answers. For this approach, I'm stuck at finding a way to let the model for the measured state make estimates w/o being influenced by bad values. However, as stated above, I'm still considering the other two approaches.

• Just to clarify, although I initialyy thought the problem seemed simple, I quickly found it to be complicated. I feel somewhat relieved to be informed that others feel similarly :) Commented Jun 2, 2013 at 1:18
• Just to be clear, you have no other data, or timestamps, that might be helpful in flagging erroneous 4.7's? Or you've looked at them and found no relationship. And it's not possible to experiment on the remote sensor? (I.e. subject it to a known, constant condition and watch for the 4.7's.) Commented Jun 2, 2013 at 15:05
• It's a time series, and the measurements are made at regular intervals. The other data isn't related to the 4.7's. 4.7's pop up under a variety of field conditions. The sensor will (hopefully) be fixed soon, but the statistical question remains. Commented Jun 2, 2013 at 16:52

I would guess that you could take every value that was 4.7 or so, and use

• Various interpolation rules to replace it. These rules could start with linear, cubic, cubic spline or piecewise cubic Hermite interpolation. On the last, see http://en.wikipedia.org/wiki/Cubic_Hermite_spline

• Any appropriate time series smoothing or forecasting method (e.g. some kind of exponential smoothing) to replace it.

If the 4.7 is genuine, a good method should replace it with about the same value any way (that's optimistic, but I don't know what other attitude to take).

How would you assess which possible method was least bad? Take genuine values not 4.7, replace them with missing, then see which method got closest, using a criterion for closest that matches your data generation process and statistical philosophy.

(I wouldn't call this a simple problem. Certainly my suggested solution isn't simple.)

From your description you have no way to be certain if a particular observation of 4.7 is genuine or not; if the series is near 4.7 you could have any number of 'legitimate' 4.7's mixed with any number of 'bad' ones.

As such, any assessment would probably be best regarded probabilistically.

I'd be inclined toward a Bayesian (indeed, MCMC) approach that has a model for the generation of the 'bad' 4.7-s and (from the model for the rest of the data) is able to infer the lost information on any iteration it decides an observation near 4.7 is bad.

You'd end up with some points near 4.7 being always or almost always regarded as 'bad' (the ones where it's obviously bad) and then it's treated like missing data, and some points near 4.7 that are mostly regarded as 'good', which are occasionally replaced with a value suggested by the other data and your model (which will also turn out to be more or less near 4.7), and some data that's 'in between' (where you might suspect it could be bad but wouldn't be sure), which is often but not always replaced.

• I agree strongly that this is not "a fairly simple problem"! I wrote something like that in an early draft of my answer. What can be inferred from the rest of your data, if there are any, is a crucial aspect I did not address. If my suggestion has merit, it's that it is much simpler and easier to implement than Glen_b's more advanced suggestion. Commented Jun 1, 2013 at 9:08
• I made an edit to my question to elaborate on an approach I'd be contemplating. It's very similar (in my opinion, but I'd be happy to get more feedback) to what @NickCox and Glen_b suggested. Commented Jun 2, 2013 at 1:21