I have a system that makes predictions in real time. Whenever the system encounters a positive label, the process must be stopped. This, however, means that noisy predictions, such as the one shown at time 10, is a false positive that should be removed, in favor of the later, more certain predictions.
As I'm evaluating the performance offline on already-happened data, I'm afraid to introduce future bias. Consider, for example, some sort of forward-moving filter. Even in a short timespan, several filters would contain future information, so these are obviously no-go.
How can I smoothe and evaluate in a way that mimics future unseen data?
import matplotlib.pyplot as plt import numpy as np p = np.random.normal(0, 0.1, 30) p = p + 1 p[20:] = p[20:] + 1 p = (p - min(p)) / (max(p) - min(p)) fig, axes = plt.subplots(nrows=2, sharex=True) axes.plot(p) for start in range(30 - 5): end = start + 6 c = "red" if start < 10 and end > 10 else "black" axes.plot([start, end], [start, start], color=c, alpha = 0.5) for ax in axes: ax.axvspan(0, 20, alpha = 0.1, color="red") ax.axvspan(20, 30, alpha=0.1, color="green") ax.axvline(10, color="black", ls=":") plt.show()