Suppose I am measuring 3 variables over time in 5 minute intervals. These variables are measuring the same thing, but using different metrics. My goal is to determine which one of these metrics is the best. When “unusual” behavior occurs one (or many) of the signals catches it, but the others don’t. Does there exist a statistical way to quantify how well one signal does compared to others based on the other’s signals inability to catch certain “unusual” behavior?
I could just do a count of the number of “unusual” events that are caught by each signal and choose the one with the greatest output, but what if my sample of time frame does not contain data that is representative of what might happen in the future?
As you can see, I am at a very rudimentary level of this problem and some suggestions/ideas would be greatly appreciated!