5
$\begingroup$

My understanding of the mutual information between two random variables X and Y can be stated formally as follows:

$$I(X ; Y) = H(X) — H(X | Y)$$ Where $I(X; Y)$ is the mutual information for $X$ and $Y$, $H(X)$ is the entropy for $X$, and $H(X | Y)$ is the conditional entropy for $X$ given $Y$. The result has the units of bits(zero to one).

My question on mutual_info_regression in scikit-learn is whether (1) it is returning the "marginal" mutual information between each feature of $X$ and $Y$, i.e. $I(X_1 ; Y)$, $I(X_2 ; Y)$, ... $I(X_n ; Y)$ etc, or (2) it is actually returning the mutual information between $X_1$ and $Y$ taking into account of other features $X_2,...X_n$ and mutual information between $X_2$ and $Y$ taking into account of other features $X_1,X_3...X_n$ ?

I believe it should be the former (1), as I tested the function with code below. However, I would like to be sure I am understanding mutual_info_regression correctly before using it further. My goal is to compute mutual information between a set of features ${X_1,...X_n}$, which would form a $n \times n$ matrix. If mutual_info_regression is returning (1), then i only have to run mutual_info_regression $n$ times to fill the entire matrix instead of running it $n^2$ times.

import pandas as pd
from sklearn.feature_selection import mutual_info_regression
import numpy as np

df= pd.DataFrame(np.random.normal(10,10,[100,11]))
X = df.iloc[:,0:9]
Y = df.iloc[:,10]

mutual_info_regression(X,Y)
mutual_info_regression(X[1].to_frame(),Y)
mutual_info_regression(X[2].to_frame(),Y)

I am asking the question above because the term regression confuses me here. In normal regression, the regression coefficient represents the difference in the predicted value of the response variable for each one-unit change in the predictor variable, assuming all other predictor variables are held constant.

$\endgroup$

1 Answer 1

3
$\begingroup$

(1) is correct. mutual_info_regression calculates the mutual information between each feature of $X$ and the target variable $Y$ independent of the remaining features of $X$.

The relevant code in the function is this part:

mi = [
    _compute_mi(x, y, discrete_feature, discrete_target, n_neighbors)
    for x, discrete_feature in zip(_iterate_columns(X), discrete_mask)
]

(Source Line 304-307)
X is the feature matrix and discrete_mask is a boolean array indicating which features are discrete and which are continuous. The code iterates over each single column and independently calculates the mutual information between it and the target y.

scikit-learn just uses the words classification and regression to differentiate between the type of target variable.
This is further reflected in the code linked above where the single major difference between the two functions is that mutual_info_classif sets discrete_target to True while mutual_info_regression sets it to False.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.