Forgive my ignorance, I am not a mathematician or statistician. I'll try to explain as clearly as I can.
I have a dataset (image below). The data represents delays of some signal over time. The large "spikes" are caused by incorrect predictions - when an incorrect prediction is made, the delay increases sharply. However, noise and whatnot causes some smaller spikes to occur between the larger ones.
I want to define a metric which will describe how "stable" or "jumpy" my data is, if that makes any sense.
Variance tells us how far values are from the mean, but in my case the large spikes will give too much weight and not give me any information about the smaller spikes. I thought about if I normalise my values and then calculate the variance... but I'm not sure... I've also read about the "index of dispersion", but I cannot tell if this is what I'm after. It's all very confusing to me!
Consider this graph:
And compare it to:
Graph 1 is far more stable: both encounter incorrect predictions (large spikes), but when predictions are correct (between the large spikes), there is a clear difference between the two graphs.
I thought that perhaps this is what I can do:
- normalise all data points (I want a score between 0 and 1)
- Sum the normalised values
- divide the sum by the number of data points.
Like in this Python 3 program:
import random
vector = [int(100*random.random()) for i in range(10)]
print(vector)
min_value = min(vector)
max_value = max(vector)
normalised = [(value-min_value)/(max_value-min_value) for value in vector]
print(normalised)
answer = sum(normalised) / len(normalised)
print(answer)
My question is: Have I done this correctly, or is there another/better way to do this?