How can feature relevance be assessed with categorical data?

Question

THE PROBLEM

Consider data such as

   feature_1   feature_2  feature_3   feature_4
0   0.192812  f2_class_1   0.274992  f4_class_1
1   0.456625  f2_class_0   0.284048  f4_class_2
2   0.989948  f2_class_0   0.194613  f4_class_0
3   0.233459  f2_class_0   0.646692  f4_class_0
4   0.107654  f2_class_1   0.281131  f4_class_1

where features 1 and 3 are numerical and features 2 and 4 are categorical. Assuming that the features 2 and 4 are drawn from a pool of 2 and 3 categories, respectively, then one-hot encoding for the above data gives

   feature_1  f2_0  f2_1  feature_3  f4_0  f4_1  f4_2     
0   0.192812    0     1   0.274992    0     1     0 
1   0.456625    1     0   0.284048    0     0     1 
2   0.989948    1     0   0.194613    1     0     0 
3   0.233459    1     0   0.646692    1     0     0 
4   0.107654    0     1   0.281131    0     1     0 

So, effectively, the machine learning is done on, not 4 features, but 7: feature_1, f2_0, f2_1, feature_3, f4_0, f4_1, and f4_2.

THE QUESTION

Naively running a function such as scikit-learn's SelectKBest() will return a relevance score, say, a p-value, of the 7 features, whereas in reality, what I want is an ordering of the 4 original features. How do I go about that?

Also, assuming I want to process this data using a feed-forward neural network (aka. multi-layer perceptron), is it enough to provide it with the 7 features at the input layer or is there more processing that needs to be done to account for the fact that f2_0 and f2_1 "belong together" (and similarly for the f4's)?

Answer 1

Naively running a function such as scikit-learn's SelectKBest() will return a relevance score, say, a p-value, of the 7 features, whereas in reality, what I want is an ordering of the 4 original features. How do I go about that?

In general, feature selection is hard. For a small number of features, though, it should be computationally feasible to use mutual information methods. For an overview, see

"A review of feature selection methods based on mutual information" by Jorge R. Vergara and Pablo A. Estévez. Neural Computing and Applications, January 2014, Volume 24, Issue 1,

Also, assuming I want to process this data using a feed-forward neural network (aka. multi-layer perceptron), is it enough to provide it with the 7 features at the input layer or is there more processing that needs to be done to account for the fact that f2_0 and f2_1 "belong together" (and similarly for the f4's)?

You have a few options.

• For a categorical variable with $k$ levels, you can use binary encoding to make $k$ vectors with values $\{0,1\}$ or perhaps $\{-1,1\}$, indicating which category is "present". If you include bias neurons, this strategy introduces some redundancy to your model, though (cf "dummy variable trap" from your regression analysis textbooks). However, since neural networks are not identified in general, this is not an inherent obstacle for model estimation.

• You can keep only $k-1$ categories. This is the standard regression categorical encoding, and it avoids the dummy variable trap.

• You can use entity encoding, which is a more sophisticated network structure. It adds between 1 and $k-1$ hidden, linear neurons between the categorical input and the first fully-connected layer. This has some nice empirical results behind it.

"Entity Embeddings of Categorical Variables" by Cheng Guo, Felix Berkhahn

We map categorical variables in a function approximation problem into Euclidean spaces, which are the entity embeddings of the categorical variables. The mapping is learned by a neural network during the standard supervised training process. Entity embedding not only reduces memory usage and speeds up neural networks compared with one-hot encoding, but more importantly by mapping similar values close to each other in the embedding space it reveals the intrinsic properties of the categorical variables. We applied it successfully in a recent Kaggle competition and were able to reach the third position with relative simple features. We further demonstrate in this paper that entity embedding helps the neural network to generalize better when the data is sparse and statistics is unknown. Thus it is especially useful for datasets with lots of high cardinality features, where other methods tend to overfit. We also demonstrate that the embeddings obtained from the trained neural network boost the performance of all tested machine learning methods considerably when used as the input features instead. As entity embedding defines a distance measure for categorical variables it can be used for visualizing categorical data and for data clustering.

Answer 2

One possible approach for the feature importance problem is to use permutation importance, which is very nicely described here.

After a very brief search I can't find it implemented directly in scikit-learn, but it seems to be available in some other package.