Check if my time series is forecastable using Shannon entropy

Question

According to this answer: https://datascience.stackexchange.com/a/95232/141037, is possible to verify the forecastability of a time series using the Shannon entropy, the lower the Shannon entropy value, the more forecastable it is. My question is: the time series I use for forecasting tests (regression problem) is windowed (shape (videos, frames, frame height, frame width), I applied a sliding window to it). Can I use this windowed time series to calculate shannon entropy?

Answer 1

You can apply any technique you want to your data, the trouble will always be with interpreting the results. Say you compute entropy, how do you know which entropy is significant? How do you know one signal's entropy is substantially larger/smaller than another?

Besides analytical derivations, here one can take an empirical approach. What type of time-series are you expecting to forecast? Say something similar to ARMA with one independent variable:

$$ \begin{align} y_t-\phi \cdot y_{t-1}&=\epsilon_t-\theta \cdot \epsilon_{t-1} + \eta\cdot x_t \\ \left(\begin{array} \\ 1 & -\phi & 0 & 0 & \dots & 0 & 0 & 0\\ 0 & 1 & -\phi & 0 & 0 & \dots & 0 & 0\\ \vdots & \ddots & \ddots & \ddots & \ddots & \dots & 0 & 0\\ \vdots & \ddots & \ddots & \ddots & \ddots & \dots & 0 & 0\\ \vdots & \ddots & \ddots & \ddots & \ddots & 1 &-\phi & 0\\ 0 & 0 & 0 & \ddots & \ddots & \dots & 1 & -\phi\\ 0 & 0 & 0 & 0 & 0 & \dots & 0 &1 \\ \end{array} \right) \left(\begin{array} \\ y_t \\ y_{t-1} \\ \vdots \\ \vdots \\ \vdots \\ \vdots \\ y_2 \\ y_1 \end{array} \right)&= \left(\begin{array} \\ \epsilon_t-\theta\cdot \epsilon_{t-1}+\eta\cdot x_t \\ \epsilon_{t-1}-\theta\cdot \epsilon_{t-2}+\eta\cdot x_{t-1} \\ \vdots \\ \vdots \\ \vdots \\ \vdots \\ \epsilon_2-\theta\cdot y_1+\eta\cdot x_2 \\ \epsilon_1+\eta\cdot x_1 \end{array} \right) \end{align} $$

Where $\epsilon_{\dots}$ is your noise, $x_{\dots}$ is the independent variable, $y_{\dots}$ is dependent (and is zero for $t\le 0$). One can re-arrange the above to:

$$ \left(\begin{array} \\ y_t \\ y_{t-1} \\ \vdots \\ \vdots \\ \vdots \\ \vdots \\ y_2 \\ y_1 \end{array} \right)= \left(\begin{array} \\ 1 & \phi & \phi^2 & \phi^3 & \dots & \phi^{n-1} & \phi^{n-2} & \phi^{n-1}\\ 0 & 1 & \phi & \phi^2 & \phi^3 & \dots & \phi^{n-1} & \phi^{n-2} \\ \vdots & \ddots & \ddots & \ddots & \ddots & \dots & \dots & \dots\\ \vdots & \ddots & \ddots & \ddots & \ddots & \dots & \dots & \dots\\ \vdots & \ddots & \ddots & \ddots & \ddots & 1 &\phi & \phi^2\\ 0 & 0 & 0 & \ddots & \ddots & \dots & 1 & \phi\\ 0 & 0 & 0 & 0 & 0 & \dots & 0 &1 \\ \end{array} \right) \left(\begin{array} \\ \epsilon_t-\theta\cdot \epsilon_{t-1}+\eta\cdot x_t \\ \epsilon_{t-1}-\theta\cdot \epsilon_{t-2}+\eta\cdot x_{t-1} \\ \vdots \\ \vdots \\ \vdots \\ \vdots \\ \epsilon_2-\theta\cdot y_1+\eta\cdot x_2 \\ \epsilon_1+\eta\cdot x_1 \end{array} \right) $$

Where we assume $n$ time-steps. So all you need to do now is generate random $\epsilon$, choose $\theta,\phi$ and $x_t$. Then plug everything into the vector on the right, multiply it by the matrix (formally the inverse of a Lag Operator matrix). And you will get time-series that is correlated as per your spec. You can then apply all the procedures you want on it, and test to see whether those procedures will allow you to distinguish between predictable (mostly controlled by large $\eta$ and $\phi$) and un-predictable time-series. You can even get an empirical distribution of your entropy (by repeating this process many times.)

This is just one example of course. There are other ways you can generate time series. This one is nice because it is analytically tractable