Predictability of a time series

Question

Say we are given a time series $(x_t)_{t \in P}$ where $P$ is the index set of past observations (train set).

Imagine that we have built a model for our data and now want to assess predictability of the time series. To this end suppose we are given $(x_t)_{t \in F}$ where $F$ is the index set of future observations (test set). I am interested in finding a good definition of predictability (relative to the model) in this context.

For $t\in F$ denote $\hat x_t$ the prediction of $x_t$. A natural performance metric would be: \begin{align*} MSE = \sqrt{\frac{1}{|F|}\sum_{t\in F} |\hat x_t - x_t|^2}. \end{align*} Now, the idea would be to compare this performance of the model with the performance on shuffled data $(s_t)_{t\in F}$ where $s_t = x_{\sigma(t)}$ and $\sigma$ is a random permutation of $F$, i.e. define the quantity: \begin{align*} \alpha = 1-\sqrt{\frac{\sum_{t\in F} |\hat x_t - x_t|^2}{\sum_{t\in F} |\hat x_t - s_t|^2}}. \end{align*}

In other words, $\alpha$ is the quotient of the MSE of "normal" data over the MSE of "shuffled" data. $\alpha \approx 0$ implies $\sum_{t\in F} |\hat x_t - x_t|^2<<\sum_{t\in F} |\hat x_t - s_t|^2$ suggesting that the model captured some structure in the series (high predictability), and $\alpha \approx 1$ implies $\sum_{t\in F} |\hat x_t - x_t|^2 \approx \sum_{t\in F} |\hat x_t - s_t|^2$ suggesting that the model capture no structure at all in the series (low predictability).

By playing around with toy models of the form $x_t = f(d_t) + e_t$ where $d_t$ is a deterministic sequence and $e_t$ are i.i.d. $\mathcal{N}(0,\tau^2)$, computing $\alpha$ for different calibrations of $\tau$ result in a clear increasing curve from 0 to 1 where the small values corresponds to small values of $\tau$ and vice and versa. There is a good alignement with the intuition of predictability in this case when we compare the value of $\alpha$ and the shape of the series. But does this make sense at all? How would one declare that $\alpha$ is small enough to declare unpredictability or high enough to declare predictability?

Answer 1

Your $\alpha$ reminds me of common relative point forecast accuracy metrics. Such metrics are calculated by taking the error of your focal method and dividing it by the error achieved by some benchmark method, like a naive forecast. The difference is that your $\alpha$ compares one model's performance on your actual data against its performance on randomized data, whereas relative error metric compare different models on the same evaluation data.

I think your $\alpha$ might be an interesting way of looking at time series. The problem is that it evaluates time series in the context of a model. (Relative error metrics do the reverse: they evaluate a model, or rather a forecast, in the context of evaluation data.) If the model is misspecified, like non-seasonal for seasonal data, or not including important causal drivers, then the relationship of $\alpha$ to a "predictability" construct will be off. We have a popular thread on How to know that your machine learning problem is hopeless? that may be helpful.

In the end, I don't think that there will be useful general cut-offs to declare predictability or non-predictability. Some things are very easy to predict, like the time the sun will rise tomorrow. Some things are harder to predict, like supermarket sales (which I do). Other things yet are very hard to predict, like a roll of a standard die, but we may at least have a probabilistic idea of what will happen, so we can hedge the uncertainty. So it really boils down to whether something is predictable enough for what you plan on doing with the forecast, i.e., for the decision you plan on taking based on the forecast.