What is the Mutual Information for one instance?

Question

I'm computing mutual information for several features where one of my datasets has one instance. One instance is because of a specific filtering criterion I used. I'm using sklearn.feature_selection.mutual_info_classif¶ to calculate MI. I am getting an ValueError due to this one instance. Anyway, I wanted to quantify this case MI. Therefore I was looking for the definition of MI and searching for ways to quantify this specific case.

Is it 0 ? - meaning just one feature value and label, no information content?

How do we quantify this specific situation? Any suggestions are appreciated.

---

Example:

X = array([[7., 7., 0., 0., 1., 0., 2., 1., 0.]]) #set of feature values
Y = array([2.]) #label
mi_scores = mutual_info_classif(X, Y,random_state=0)  

Something like the above. Then I'm getting ValueError: Found array with 0 sample(s) (shape=(0, 1)) while a minimum of 1 is required.

Answer 1

There are two problems with your programming.

1. Y has to have length n(as in number of samples), so if it is constant, repeat it.
2. X separates the individual samples by column. You separated them by row. You can fix that by transposing it.

As a result, you get numerically 0, i.e. something like 1.11022302e-16.

Here is some example code. Note that i clearly mark my imports, which is best practice for providing examples.

import sklearn.feature_selection as sfs
import numpy as np

#set of feature values, samples separated by columns, thanks to transpose
X = np.transpose(np.array([[7., 7., 0., 0., 1., 0., 2., 1., 0.]])) 
#label, constant across observations
Y = np.array([2., 2., 2., 2., 2., 2., 2., 2., 2.]) 
mi_scores = sfs.mutual_info_classif(X, Y,random_state=0)  
print("final:", mi_scores)


#Example from Chat-GPT for 5 samples and 3 faetures
X = np.array([[1.0, 2.0, 3.0],
              [4.0, 5.0, 6.0],
              [7.0, 8.0, 9.0],
              [10.0, 11.0, 12.0],
              [13.0, 14.0, 15.0]])

# Create a sample target vector y with shape (n_samples,)
y = np.array([0, 0, 0, 0, 0])
print(sfs.mutual_info_classif(X, y,random_state=0) )