I have a paired dataset whose observed values are:

x       y
4.0     4.26
5.0     5.68
6.0     7.24
7.0     4.82
8.0     6.95
9.0     8.81
10.0    8.04
11.0    8.33
12.0    10.84
13.0    7.58
14.0    9.96

From these observed values, I can compute correlation to be 0.816 between $x$ and $y$.

How can I transform the observed values into probabilities $p(x)$ and $p(y)$ so that I can also compute the mutual information score between these probabilities of $x$ and $y$? Since each value is unique in this example, does that just mean each observed value has an equal probability of occurring, therefore, we use a uniform vector of $1/N$s for $p(x)$ and $p(y)$, where $N=11$ is the length of the two vectors?

p(x)    p(y)
1/11    1/11
1/11    1/11
1/11    1/11
1/11    1/11
1/11    1/11
1/11    1/11
1/11    1/11
1/11    1/11
1/11    1/11
1/11    1/11
1/11    1/11

Afterwards, how can we comparatively interpret the resulting correlation and mutual information estimates if one is derived from observed values, and the other from probabilities of the observed values? If correlation is 0.816 and mutual information comes out be either higher or lower than that number, what new statistical inference has been acquired by looking at mutual information?

  • 2
    $\begingroup$ Nobody observes probabilities directly: probabilities are hypothetical quantities in mathematical models. Thus, there's never a question of "transforming" data to probabilities: you have to estimate the probabilities from the data. That requires you to specify a family of models for the data. $\endgroup$
    – whuber
    Aug 5, 2020 at 21:00
  • 2
    $\begingroup$ Estimating MI for continuous variables can be tricky. There are several methods for performing such estimation. One simple approach is binning the data and then move to a discrete pmf (a naive approach I would avoid). Other methods are based on kernel density or nearest-neighbors estimators. Surely, you cannot just transform the observed values into frequencies (probability of randomly choosing two identical numbers in $\mathbb{R}$ is 0). $\endgroup$
    – osmoc
    Aug 5, 2020 at 21:01
  • $\begingroup$ Can someone give me an example of what the p(x) and p(y) vectors look like then with their chosen technique, or whichever method a statistician would gravitate towards by default? $\endgroup$
    – develarist
    Aug 9, 2020 at 15:29

1 Answer 1


You can read about some options for the "mutinformation" function in the R package "infotheo"

You have to choose an estimator in order to do the transformation:

"emp" : This estimator computes the entropy of the empirical probability distribution.
"mm" : This is the Miller-Madow asymptotic bias corrected empirical estimator.
"shrink" : This is a shrinkage estimate of the entropy of a Dirichlet probability distribution.
"sg" : This is the Schurmann-Grassberger estimate of the entropy of a Dirichlet probability distribution.



Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.