I am working on a machine learning algorithm which performs a binary classification. I have different features, and I would like to know which of them are better for the classification, I mean what of them really make a difference between classes and what of them are not important.

I have thought of computing the normal distribution of every feature in class A and B and getting the overlap between them. If they have a big area in common the feature is not good.

I have also heard about Principal Component Analysis. I do not know what is the best way of determining which are the best features, so I ask or your help.

Thanks a lot!!!!


3 Answers 3


As I continue to investigate this subject I found out that there are different methods to compare features, I am going to post some of them, just in case someone has the same problem than me:

1.-Kolmogorov-Smirnov compares the maximum distance between two cumulative distribution functions and returns a value meaning the similarity between the functions. If you compare the feature from sample A and B gives you an idea, if they are close is not a good feature.

2.-Compute overlap between distribution functions, the bigger the overlap the worse the feature is to differenciate the classes, code:

# compute overlap between the 2 distributions
ker_sick = stats.gaussian_kde(sick)
ker_healthy = stats.gaussian_kde(healthy)
min_point, max_point = aux.get_min_max(sick, healthy)
points_range = np.linspace(min_point, max_point, 100)
sick_points = ker_sick(points_range)
healthy_points = ker_healthy(points_range)
min_points = aux.min_between_two_list(sick_points, healthy_points)

def y_pts(pt):
    y_pt = min(ker_sick(pt), ker_healthy(pt))
    return y_pt

overlap = integrate.quad(y_pts, a=-np.inf, b=np.inf)
print("overlap: ", overlap)

# plot distributions (healthy, sick) and the overlap between them
fig = plt.figure()
ax = fig.add_subplot(121)
ax = sns.kdeplot(sick, shade=True, cut=0, label="healthy", color='g')
ax = sns.kdeplot(healthy, shade=True, cut=0, label="sick", color='r')
ax.plot(points_range, min_points, color="y", alpha=1)
ax.fill_between(points_range, 0, min_points)
ax.set(xlabel='value', ylabel='probability', title=name)

3.- Previous methods are bit more homemade, scikit also includes some module for feature selection using different methods to compare features and let you know which one is better, Scikit feature selection

I tried these 3 methods, including several variatons of the scikit feature selection, and they always matched in which was the best feature, so I guess everyone of them is working properly.

Thanks a lot and I hope this ends up being helpful!


First off, principal component analysis is not necessarily what you want. PCA is a method of dimensionality reduction, meaning that it reduces the dimensions of your data. This may be good for you, but you should know that the algorithm will change the value of the components in each dimension. See this: http://setosa.io/ev/principal-component-analysis/

Your thought about distributions is a very good idea! So good, actually, that staticians have been doing something similar for a long time. Try computing the Kullback-Leibler divergence of the distributions of your two classes for each dimension in your data. Don't just do it, understand what it means: https://www.countbayesie.com/blog/2017/5/9/kullback-leibler-divergence-explained or look at my own post https://github.com/abrhor/Momentum-Neural-Networks/blob/master/Distributions/Conditional%20Distribution%20of%20RSI%20Notebook.ipynb which is not as informative and also has some finance and a lot of code.

KL divergence will give you information about your data, but won't help simplify the computation of the actual classifier like PCA does:

Finally, just a convention, you don't mean computing the normal distributions. "Normal" is a term for a regular distribution/histogram if the plot is or is approximately symmetrical, because it has certain resulting statistical characteristics.

  • $\begingroup$ thanks a lot for your answer, now I know what PCA does and it is not what I am looking for, I only want to compute the "distance" between two distributions. I have been reading the info you posted about KL and I see a problem, they say "It may be tempting to think of KL Divergence as a distance metric, however we cannot use KL Divergence to measure the distance between two distributions." so I think it is more intended to compare the similarity of x classes to a given one, not the distance between two of them. You are totally right, PCA gives me a dimensionality reduction that I do not want. $\endgroup$
    – Kailegh
    May 30, 2017 at 22:06
  • $\begingroup$ I am still looking for more info, but this seems to be similar to what I am looking for: "en.wikipedia.org/wiki/Kolmogorov%E2%80%93Smirnov_test" and python implementation: "docs.scipy.org/doc/scipy-0.14.0/reference/generated/…" $\endgroup$
    – Kailegh
    May 31, 2017 at 9:55

I am no expert in machine learning, but there is a method similar to what you have described called the "Receiver Operator Characteristic" (ROC), which evaluates the the ability of a continuous variable to distinguish between binary outcomes.

Bellow I have linked a video that carefully explains how the ROC curve works, as well as the Wikipedia page, which gives a good description of the method.



The ROC curve is easy to request in most statistical software packages. I have personally used it more often in behavioral and social sciences, but I have been told that it is quite useful in machine learning. The area under the ROC curve is a decent metric of how well a factor is able to classify your binary variable, so you could compare separate factors or models using this metric.

  • $\begingroup$ As far as I know, ROC curve is used to compare True Positives against False Positives for a certain classifier, not for a certain feature, and the graphic is obtained by changing the threshold. So, I may be wrong but I think ROC curve is not what I am looking for $\endgroup$
    – Kailegh
    May 30, 2017 at 11:15
  • $\begingroup$ Alright. You likely know better than I. I hope you find what you're looking for :) $\endgroup$ May 30, 2017 at 11:24

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.