Using Mahalanobis distance for feature selection in NLP

Question

I want build a classifier that classifies sentences into two categories, and for that I have a training set of 1000 labeled sentences. My features consist of a list of about 8000 words, and for each sentence I measure the word frequency (feature divided by total words in the sentence).

I've been reading up on the Mahalanobis distance, but I haven't yet managed to understand how I could apply it to my problem. How can I use it to select the features that work best for distinguishing between the two categories?

Thank you!

Answer 1

You can use the Mahalanobis distance to quantify the distance of features of sentence $S_1$ to sentence $S_2$. Using this distance $d(S_1,S_2)$, you can apply the k-means algorithm that is suitable also for classification. For k-means classification, see e.g. this.

Alternatively, you can apply a kernel function, i.e. $K(d(S_1,S_2))$ to this distance and learn an RBF network that would classify your sentences. To be more specific, the approach can be described in the following steps:

1. Select some (say 50) sentences from your set at random. Let us call them centers.
2. For all 1000 sentences calculate distances to the centers. Thus, you will have a 1000 x 50 distances.
3. Quantify the values of the kernels. That means, apply a kernel to each of those 1000 x 50 numbers.
4. Use a linear regression to fit the output value (index of the class either 0 or 1).