5
$\begingroup$

I have a camera that detects every time it views a car. Each detection is recorded in a database. I then simulate this behavior as a time series by doing an each hour count of the records.

The problem is that for some time intervals I have no records and it is impossible to know if it is because no cars have passed or because of some bug in the system.

So I have two options: assume that no cars have passed and give zero value to those intervals or assume errors in the system and find a function that best fits the series to impute missing values.

The question is which of the two options will be better and how do I know?

thanks in advance

$\endgroup$

1 Answer 1

6
$\begingroup$

Actually, there will be many intervals when you have no observations... namely, all the intervals between two recorded detections. You could ask the exact same question about a detection-less interval of one second as for one of two days.

I presume you are asking the way you do because you are aggregating detections into some buckets (e.g., hourly), and some of your buckets have zero counts. But even buckets with nonzero counts may have missed cars because the sensors were down for part of the bucket.

There is really no way to go about this without assumptions. Ideally, get some data of known good quality, e.g., by standing in the spot for two hours and counting cars yourself, then comparing to what your sensor detected. If this yields data that is close enough to the sensor reading, your sensor is presumably working well enough at least part of the time.

If so, I would start building a model for counts, with features time of day (spline transformed) in an interaction with day of week (since you almost certainly have ; this may be helpful), possibly holidays or weather. Calculate "backward-looking" prediction intervals. Any count that comes in low is suspect, especially if there are multiple low or zero counts in succession. (But there may also have been roadwork.) Look at all these issues, and decide either case by case, or by using some rule, what to do.

You can replace problematic values by sampling from the model's predictive distribution. Just using the fitted value will underestimate the variability (though if you only replace a small amount of data, this may not be a major issue).

$\endgroup$
2
  • $\begingroup$ Thank you very much for your answer. As you well suppose I am adding in buckets every one hour. From what I understand it is better to replace the problematic values by sampling from the predictive distribution of the model than to assume zero values. In my case I would have to replace 25% of the values at most. $\endgroup$ Commented Feb 22 at 17:06
  • 1
    $\begingroup$ Hm. If 25% of your data need to be replaced, that is a lot. I can't help wondering how well you can detect the problematic cases, and whether the model you use to impute is not in itself tainted by problematic data. I would recommend that you do a sensitivity analysis and check whether your final results depend strongly on the cleansing and imputation step. $\endgroup$ Commented Feb 22 at 22:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.