How to correctly re-formulate mutual information between time series? - Cross Validated most recent 30 from stats.stackexchange.com 2019-09-23T07:07:12Z https://stats.stackexchange.com/feeds/question/316224 https://creativecommons.org/licenses/by-sa/4.0/rdf https://stats.stackexchange.com/q/316224 3 How to correctly re-formulate mutual information between time series? user2780519 https://stats.stackexchange.com/users/186597 2017-11-29T03:15:33Z 2017-12-05T22:59:46Z <p>I am trying to understand the re-formulation of mutual information between time series presented in <a href="http://www.ieice.org/proceedings/NOLTA2005/HTMLS/paper/7111.pdf" rel="nofollow noreferrer">Galka et al. </a> </p> <p>They note "If the data is given as a pair of time series \$x_t\$ and \$y_t\$, \$t = 1, . . . ,N\$, the assumption of independent sampling will typically be invalid since most time series display serial correlations. Consequently we have to regard \$x_t\$ and \$y_t\$ as different random variables for each value of \$t\$ ...' They then give:</p> <p>\$I(x,y) = log(p((x_1, y_1), .. ,(x_n,y_n))) - log(p(x_1,.., x_n)p(y_1,..,y_n))\$</p> <p>Equations \$1)\$ and \$3)\$-\$24)\$ in the paper are reasonably clear to me, but I am having trouble understanding how to derive the above formulation. Specifically, it seems to be missing \$p((x_1,y_1),..,(x_n,y_n))\$ such that:</p> <p>\$I(x,y) = p((x_1,y_1),..,(x_n,y_n)) * (log(p((x_1, y_1), .. ,(x_n,y_n))) - log(p(x_1,.., x_n)p(y_1,..,y_n)))\$</p> <p>Based on the wikipedia definition of <a href="https://en.wikipedia.org/wiki/Entropy_(information_theory)" rel="nofollow noreferrer">Shannon Entropy</a>, where: </p> <p>\$H(X) = E[I(X)]\$</p> <p>All I can think of is maybe they are assuming that the \$p((x_1,y_1),..,(x_n,y_n))\$ term is equal to \$1\$?</p> <h2>EDIT:</h2> <p>After emailing an author (many thanks Dr. Galka), I was sent <a href="http://www.andreas-galka.de/galka_etal_jstatphys_2006.pdf" rel="nofollow noreferrer">this paper</a> and ".. equation 2 is conceptually the same as the second line of equation 1, except for omitting the averaging. By treating each value of the time series as a different random variable, x and y attain the dimension of the entire time series. Then the reason for omitting the averaging is that it simply becomes impossible since we only have a single time series (if we could have several time series, we could do it)."</p> <p>What I <em>think</em> this means is that \$p((x_1,y_1),..,(x_n,y_n))\$ is about finding the joint probability distribution of each of the independent pairs, \$(x_n, y_n)\$. Every time step of \$x\$ or \$y\$ would have its own probability distribution, such that: \$p(x_n, y_n) = p(x_n)p(y_n)\$.</p> <p>Thus (in this case): \$p((x_1,y_1),..,(x_n,y_n)) = \prod_{i=1}^{n} p(x_i,y_i) = \prod_{i=1}^{n} p(x_i)p(y_i)\$</p>