# Correlation uses observed values, Mutual information uses probability values

I have a paired dataset whose observed values are:

x       y
4.0     4.26
5.0     5.68
6.0     7.24
7.0     4.82
8.0     6.95
9.0     8.81
10.0    8.04
11.0    8.33
12.0    10.84
13.0    7.58
14.0    9.96


From these observed values, I can compute correlation to be 0.816 between $$x$$ and $$y$$.

How can I transform the observed values into probabilities $$p(x)$$ and $$p(y)$$ so that I can also compute the mutual information score between these probabilities of $$x$$ and $$y$$? Since each value is unique in this example, does that just mean each observed value has an equal probability of occurring, therefore, we use a uniform vector of $$1/N$$s for $$p(x)$$ and $$p(y)$$, where $$N=11$$ is the length of the two vectors?

p(x)    p(y)
1/11    1/11
1/11    1/11
1/11    1/11
1/11    1/11
1/11    1/11
1/11    1/11
1/11    1/11
1/11    1/11
1/11    1/11
1/11    1/11
1/11    1/11


Afterwards, how can we comparatively interpret the resulting correlation and mutual information estimates if one is derived from observed values, and the other from probabilities of the observed values? If correlation is 0.816 and mutual information comes out be either higher or lower than that number, what new statistical inference has been acquired by looking at mutual information?

• Nobody observes probabilities directly: probabilities are hypothetical quantities in mathematical models. Thus, there's never a question of "transforming" data to probabilities: you have to estimate the probabilities from the data. That requires you to specify a family of models for the data.
– whuber
Aug 5, 2020 at 21:00
• Estimating MI for continuous variables can be tricky. There are several methods for performing such estimation. One simple approach is binning the data and then move to a discrete pmf (a naive approach I would avoid). Other methods are based on kernel density or nearest-neighbors estimators. Surely, you cannot just transform the observed values into frequencies (probability of randomly choosing two identical numbers in $\mathbb{R}$ is 0). Aug 5, 2020 at 21:01
• Can someone give me an example of what the p(x) and p(y) vectors look like then with their chosen technique, or whichever method a statistician would gravitate towards by default? Aug 9, 2020 at 15:29

"emp" : This estimator computes the entropy of the empirical probability distribution.