2
$\begingroup$

I need to evaluate the accuracy of a prediction model (binary classification), given that I have some kind of uncertainty on the ground truth measurement.

The model predicts the occurrence of an event at time T, T+5mn, T+10mn, etc. Unfortunately, the measurements are less frequent, and not aligned. For instance, it can say that an event occurred between T+2 and T+17 (15mn period, not aligned).

If both prediction and measures are negative on a period, then I have true negatives.

But some configurations have uncertainties: if the prediction is negative at T+5,T+10,T+15, but the measurement in the range [T+2,T+17] is positive, I can deduce that at least one of the 3 predictions is a false negative, but maybe not all.

Is there a way to model this uncertainty in the evaluation of the prediction ?

A basic way would be to count the best-case and worst-case false negatives and get a range in the end (and the same for positives). Any better way ?

$\endgroup$
4
  • $\begingroup$ In your example, what if the event took place at T+16 or T+17? Then wouldn't you have true negatives? $\endgroup$ Commented Sep 30, 2022 at 10:16
  • $\begingroup$ If the event occurs at T+16 or 17, it would be measured by the sensor (positive), and yes I would have uncertainty for all 3 predictions. $\endgroup$
    – mathieu
    Commented Sep 30, 2022 at 15:45
  • $\begingroup$ Do I understand correctly, that your prediction times and measurement times do not coincide, but you believe that values at nearby times are likely to be equal? And now you think about aggregating the measurements over intervals (periods)? What do you mean by "the measurement in the range [T+2,T+17] is positive"? Does it mean that the average of measurements in this interval is positive? And how do you determine those interval borders? $\endgroup$
    – frank
    Commented Oct 2, 2022 at 7:59
  • $\begingroup$ The phenomenon has a duration in time. The measurement is done by a in-field sensor, doing an integral over time. The time boundaries of the sensors depends on external factors. The models predicts an intensity of the phenomenon at periodic points in time. $\endgroup$
    – mathieu
    Commented Oct 3, 2022 at 11:44

1 Answer 1

1
$\begingroup$

It sounds like the uncertainty in your ground truth stems primarily from uncertainty about the true duration and intensity of the phenomena's presence: when the sensors show positive, it's not clear whether that means the phenomena has been present at lower intensities for a longer spans or at higher intensities for shorter spans.

Given that uncertainty, one option would be to systematically evaluate the model's accuracy over a broad range of scenarios, including some where the ground truth reflects short bursts of high intensity and others where the ground truth reflects longer periods of lower intensity. These alternatives could be captured by smoothing the observed binary sensor values over shorter or longer windows of time. Rolling averages provide one simple way to accomplish such smoothing. If the temporal window over which one averages remains small (zero smoothing being the smallest), then one effectively assumes high intensity inputs over the entire [T+2, T+17] interval. If, on the other hand, the window over which one averages becomes as long or longer than the interval itself, then one is assuming that the phenomena operated at lower intensity but over a longer period of time.

I don't know how much information you have about the "external factors" influencing the intensity of the phenomena and its likely durations, so I can't say what might constitute a reasonable range of window sizes to try. But examining the minimum and maximum durations during which the sensors showed positives might help.

In any event, I think you might be able to account for your uncertainty about the true inputs by evaluating your model's accuracy under a variety of plausible duration/intensity scenarios, which could, in turn, provide you with a reasonable range of possible accuracies.

$\endgroup$
2
  • $\begingroup$ that's an insightful answer thank you $\endgroup$
    – mathieu
    Commented Oct 7, 2022 at 12:39
  • $\begingroup$ @mathieu approve my answer? $\endgroup$ Commented Oct 11, 2022 at 3:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.