I have a system that makes predictions in real time. Whenever the system encounters a positive label, the process must be stopped. This, however, means that noisy predictions, such as the one shown at time 10, is a false positive that should be removed, in favor of the later, more certain predictions.

As I'm evaluating the performance offline on already-happened data, I'm afraid to introduce future bias. Consider, for example, some sort of forward-moving filter. Even in a short timespan, several filters would contain future information, so these are obviously no-go.

How can I smoothe and evaluate in a way that mimics future unseen data?

enter image description here

import matplotlib.pyplot as plt
import numpy as np

p = np.random.normal(0, 0.1, 30)
p[10] = p[10] + 1
p[20:] = p[20:] + 1
p = (p - min(p)) / (max(p) - min(p))

fig, axes = plt.subplots(nrows=2, sharex=True)

for start in range(30 - 5):
    end = start + 6
    c = "red" if start < 10 and end > 10 else "black"
    axes[1].plot([start, end], [start, start], color=c, alpha = 0.5)

for ax in axes:
    ax.axvspan(0, 20, alpha = 0.1, color="red")
    ax.axvspan(20, 30, alpha=0.1, color="green")
    ax.axvline(10, color="black", ls=":")


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.