# How to determine which are the best predictor variables given a known classification?

Given the dataset below:

RockType  Porosity  ShearFactor SandContent WaterContent PoissonCoef
1        0.10       0.78        0.06        0.19         0.77
1        0.14       0.71        0.03        0.25         0.76
1        0.10       0.73        0.05        0.17         0.76
1        0.08       0.75        0.09        0.11         0.76
2        0.23       0.72        0.08        0.28         0.75
2        0.27       0.66        0.13        0.12         0.76
2        0.29       0.64        0.11        0.09         0.74
2        0.25       0.67        0.12        0.07         0.73


I'd like to know which variables are most relevant in classifying rock samples in Rock 1 and Rock 2. Porosity seems an obvious choice given that simple example. But real-world data sets are not quite simple. Is there a method so I can perform some kind of relevance test?

Well, the first and most simple thing you can do is to compute a good old correlation matrix, to find which variables have a higher linear correlation with the one you want to predict.

From the data you provided, in seems that not only the Porosity is linearly correlated with the rock type, but also the SandContent.

A more interesting question, however, is which variables would result in a higher test accuracy if provided to a given classifier. For instance, maybe the ShearFactor has no linear correlation with the rock type, but perhaps some nonlinear transformation of this feature is super informative about the rock type, and maybe the right classification model can find that source of information.

In general, there is no straightforward way to know this without trying. And the best performing features might depend on the classification model. Nevertheless, a common approach is to use the feature "importance" as determined by some tree ensemble classification model. For instance, check out sklearn's RandomForestClassifier and its feature_importance_ attribute.

Nevertheless, you will need far more data to get any meaningful results.

• The correlation matrix seems to be a good trick to start the investigation, but as yourself already warned, it only measures linear correlation. I did crossplots between the several variables (the complete data set) and they show "crazy" relations with shapes other than ax+b that yield low correlations. Someone told me to try PCA. Will it help? I found it strange since PCA decorrelates them, right? And I've heard people talking about Random Forests. – Paulo Carvalho Mar 27 '19 at 15:29
• Well, PCA will generate a few features which are linear combinations of the original ones, are uncorrelated and explain as much variable as possible. This is done in an unsupervised manner, so the rock type would not be used. Random Forest can be used to estimate feature importances as described in my answer. – Daniel López Mar 27 '19 at 15:55

A logistic regression model with either logit or probit link function would be the best choice for you.

You include all your possible predictive variables for classification in the model as regressor variables, as long as they are not highly correlated with each other. The explained variable is the dichotomous rock type 1 or 2.

From the estimated model you obtain a prediction which is rock type 1 or 2. The predicted rock type is then compared to the original rock type of each sample and you obtain the false positive, true postive, false negative and true negative rate of your prediction. In the model you can select variables based on underlying tests or by AIC or BIC based model selection methods.

The choice of what is "positive" and "negative" is arbitrary and depends on what you wish to declare as positive or negative. Rename it as "True 1s", "True 2s", "False 1s" and "False 2s" instead. "True 1s" are when you predicted type 1 and the original rock also was type 1. "False 1s" are when you predicted type 1, but the original rock was type 2.

For a formal estimation how well your model works these rates are compared in the receiver operating characteristic (ROC) curve which is a measure of quality for your whole model and its predictive value.

• Can you please, define "false positive, true postive, false negative and true negative" in this context? I mean: why not simply false (model predicts 1 instead of 2) or true (model predicts 1 when rock type is 1)? And what do AIC and BIC stand for? – Paulo Carvalho Mar 27 '19 at 15:11
• That dependes what you hish to declare as positive or negative. Take for example this choice (the other way round is also okay): true positive: reality type 1 and prediction type1; true negative: reality type 2 and prediction type 2; false positive: reality type 2 and prediction type 1; false negative: reality type 1 and prediction type 2 – Alex2006 Mar 27 '19 at 15:17
• And a false positive? Isn't it the same as true negative in this context? – Paulo Carvalho Mar 27 '19 at 15:18
• I mean, we can only have (model 1; data 1) = true; (model 1; data 2) = false; (model 2; data 1) = false; (model 2; data 2) = true. I guess I'm missing something. Edit: now I see. – Paulo Carvalho Mar 27 '19 at 15:20
• @PauloCarvalho "negatives" and "positives" are misleading terms in your context. Call them "True 1s" and "True 2s" and "False 1s" and "False 2s" instead, then it is more clear that "True 1s" are the 1s you predicted as 1s correctly and "False 1s" are the 1s you predicted, but which were 2s in reality. – Alex2006 Mar 27 '19 at 15:27