Measuring "turbulence" in a time series

Question

Let's say I have 2 models operating on time series data. Their predictions on the test set are as follows:

   model 1: 1 0 1 0 0 1
   model 2: 1 1 1 0 0 0
   ground:  1 1 0 0 0 1

As you can see, both models have a similar score in terms of accuracy metrics w.r.t. ground truth. But I also want a score that describes the smoothness/turbulence of the model predictions over the time series with 1 number. Essentially, this measure should provide a score that penalizes (or rewards) a time series for jumpiness/turbulence/flip-floppiness (I don't know the correct term for it, hence the question).

Assuming penalization, the metric I'm looking for would give model 1 a lower score than model 2.

I can already think of some metrics such as simple counts of transitions (normalized by sequence length). So some other nice-to-haves in this metric:

• Invariant to the magnitude of the transition
• Relatively unaffected by the length of the time series
• Asymmetric in the type of transition - A correct 1->0 penalized more than an incorrect 1->0; or 1->0 penalized more than 0->1.

I'd also appreciate any references to prior work that has used such a metric. (As well as terminology for what I should have Googled)

Answer 1

Your list of three nice-to-haves does not call for anything more complicated than the transition count (or rather transition frequency). When devising a metrics, it's nice to follow a parsimony principle: the simpler the definition, the better. First, it somewhat protects you from getting a false-positive (more complex measures have more assumptions and free parameters built in, so it's easier to "fine-tune" it to yield a positive result). Second, they are easier to explain.

But conceivably, if a simple transition frequency would not work, you could also look at the statistics of inter-transition intervals. If you have examples of ground truth data, you may be able to learn something about how these intervals are distributed: is it a binomial process? (Or, if the transitions are really infrequent compared to your time step, can it be approximated by a Poissonian process?). Or is the distribution skewed because of either bursting, or a tendency for these transition events to be periodical? Are consecutive intervals independent?

If you google for tests on point processes you can find lots of oldschool analytical solutions, and then there are also modern bootstrapping techniques in which you reshuffle events one way or another, and then compare properties of these surrogate traces to that of the original trace.

And once you know something about the statistics of transitions for real data, you can punish your reconstructions not just based on the number of transitions, but based on the statistical distribution of these transitions. Because I don't know much about your application it is hard for me to fantasize further, but I think this line of thought can be cast into something practical rather easily.