1
$\begingroup$

I am playing with mutual information in scikit-learn.

import numpy as np
from sklearn.feature_selection import mutual_info_regression

X = np.linspace(-10,10,101)
y = X**2
print('MI:', mutual_info_regression(X.reshape(-1, 1),y)[0])

This gives me the result

MI: 1.759

Then I change the number of points, 

X = np.linspace(-10,10,1001)

I get 

MI: 3.918

For 10001 points: 

MI: 6.189

For 100001 points:

MI: 8.483

Why is that? Should it really depend on the number of points or I am doing something wrong? Mutual information is a measure of the mutual dependence between the two variables. I would expect from such a measure some fluctuation when the number of points is small. But here, it looks like it monotonously increases with the number of points.

Improve this question
$\endgroup$
1
  • $\begingroup$ I'm not familiar with that particular software implementation, but I'm guessing that what is happening is that the entropy is changing as you add more data because it is categorizing one or both of the inputs rather than treating them as truly discrete. Your variables are extremely dependent on one another so your MI estimate is going to be near the entropy. $\endgroup$
    – Bryan Krause
    Dec 27, 2018 at 20:41

0

Reset to default

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.