I am trying to understand the re-formulation of mutual information between time series presented in Galka et al.

They note "If the data is given as a pair of time series $x_t$ and $y_t$, $t = 1, . . . ,N$, the assumption of independent sampling will typically be invalid since most time series display serial correlations. Consequently we have to regard $x_t$ and $y_t$ as different random variables for each value of $t$ ...' They then give:

$I(x,y) = log(p((x_1, y_1), .. ,(x_n,y_n))) - log(p(x_1,.., x_n)p(y_1,..,y_n))$

Equations $1)$ and $3)$-$24)$ in the paper are reasonably clear to me, but I am having trouble understanding how to derive the above formulation. Specifically, it seems to be missing $p((x_1,y_1),..,(x_n,y_n))$ such that:

$I(x,y) = p((x_1,y_1),..,(x_n,y_n)) * (log(p((x_1, y_1), .. ,(x_n,y_n))) - log(p(x_1,.., x_n)p(y_1,..,y_n)))$

Based on the wikipedia definition of Shannon Entropy, where:

$H(X) = E[I(X)]$

All I can think of is maybe they are assuming that the $p((x_1,y_1),..,(x_n,y_n))$ term is equal to $1$?


After emailing an author (many thanks Dr. Galka), I was sent this paper and ".. equation 2 is conceptually the same as the second line of equation 1, except for omitting the averaging. By treating each value of the time series as a different random variable, x and y attain the dimension of the entire time series. Then the reason for omitting the averaging is that it simply becomes impossible since we only have a single time series (if we could have several time series, we could do it)."

What I think this means is that $p((x_1,y_1),..,(x_n,y_n))$ is about finding the joint probability distribution of each of the independent pairs, $(x_n, y_n)$. Every time step of $x$ or $y$ would have its own probability distribution, such that: $p(x_n, y_n) = p(x_n)p(y_n)$.

Thus (in this case): $p((x_1,y_1),..,(x_n,y_n)) = \prod_{i=1}^{n} p(x_i,y_i) = \prod_{i=1}^{n} p(x_i)p(y_i)$


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.