Let's say I have 2 models operating on time series data. Their predictions on the test set are as follows:
model 1: 1 0 1 0 0 1 model 2: 1 1 1 0 0 0 ground: 1 1 0 0 0 1
As you can see, both models have a similar score in terms of accuracy metrics w.r.t. ground truth. But I also want a score that describes the smoothness/turbulence of the model predictions over the time series with 1 number. Essentially, this measure should provide a score that penalizes (or rewards) a time series for jumpiness/turbulence/flip-floppiness (I don't know the correct term for it, hence the question).
Assuming penalization, the metric I'm looking for would give model 1 a lower score than model 2.
I can already think of some metrics such as simple counts of transitions (normalized by sequence length). So some other nice-to-haves in this metric:
- Invariant to the magnitude of the transition
- Relatively unaffected by the length of the time series
- Asymmetric in the type of transition - A correct 1->0 penalized more than an incorrect 1->0; or 1->0 penalized more than 0->1.
I'd also appreciate any references to prior work that has used such a metric. (As well as terminology for what I should have Googled)