# How can I find the field which most affects or contributes to decision making in a machine learning algorithm?

Consider the example below. On a larger dataset, it would be fairly obvious that name and gender are not a good indicator of whether a person is an adult or a kid, and that it's age which best decides the class.

How can I use statistics or an algorithm or a function / parameter in say, R to understand which column / field most affects the class I'm trying to predict.

gender = c('M', 'F', 'M', 'F', 'M')
name   = c('John', 'Anthony', 'Mark', 'Joe', 'Will')
age    = c(10, 20, 30, 20, 17);
class  = c('Kid', 'Adult', 'Adult', 'Adult', 'Kid')

> df = data.frame(age, gender, name, class)
> df
age gender    name class
1  10      M    John   Kid
2  20      F Anthony Adult
3  30      M    Mark Adult
4  20      F     Joe Adult
5  17      M    Will   Kid

• (To whom it may concern: although any answer here will necessarily be very brief & shallow, I don't think this question is 'too broad' to be answerable.) – gung - Reinstate Monica May 29 '16 at 16:09
• What do you want to use this information for? What kind of decision making is it going to influence? There's no canonical method for this type of thing, and your choice must be influenced by your intended use. – Matthew Drury May 29 '16 at 16:52

## 2 Answers

There are many ways to do this work. It's been called, variable importance, feature selection, ranking, ... in different fields.

These are some ideas:

1. You can build a Regression model and observe the p-values of the coefficients of each variable.

2. If you have enough data, you can try Principal Component Analysis.

3. Pearson Correlation

4. Spearman Correlation

5. Kendall Correlation

6. Mutual Information

7. (regressional) ReliefF algorithm

8. Decision trees

I have used all of these in MATLAB. Unfortunately, I don't know about them in R.

I'll just give some R examples to demonstrate what @PeyM87 already mentioned, including feature filters and wrappers, and classic feature correlation (using data from the small mtcars dataset):

d <- mtcars
d$cyl <- factor(d$cyl)
d$vs <- factor(d$vs)
d$am <- factor(d$am)
d$gear <- factor(d$gear)
d$carb <- factor(d$carb)


Lets assume the target variable is cyl. You could employ classic feature selection using feature filters and feature wrappers. Here's a feature filter example (using caret univariate filters) that states the "most important" features for the target variable:

> modelSbf <- sbf(x = d[,-2], y = d[,2], sbfControl = sbfControl(functions = rfSBF, method = 'repeatedcv', repeats = 5))
> modelSbf$optVariables  "mpg" "disp" "hp" "drat" "wt" "qsec" "vs" "am" "gear" "carb"  And here a feature wrapper example (using caret recursive feature elimination), which states the actually most important features for the target variable by including model training in the process: > modelRfe <- rfe(x = d[,-2], y = d[,2], rfeControl = rfeControl(functions = rfFuncs, method = "repeatedcv", repeats = 5)) > modelRfe$optVariables
 "disp" "mpg"  "hp"   "wt"   "vs"   "carb" "drat" "qsec"


Note that those approaches give similar but not exact same results.

Alternatively, after training a model on predicting the target variable, the variable importance for some models can just be stated:

> library(caret)
> modelRpart <- train(x = d[,-2], y = d[,2], method = 'rpart', trControl = trainControl(method = 'LOOCV'))
> varImp(modelRpart)

Overall
disp  100.00
hp     91.04
mpg    86.03
wt     69.23
vs     69.07
gear    0.00
drat    0.00
qsec    0.00
am      0.00
carb    0.00


And in case your data would only have numeric features you could also employ classic correlation, PCA, and similar approaches that only work with numeric values. Here's a feature correlation example that shown correlation between features and the target variable:

> library(corrplot)
> # correlation with target variable
> cor(mtcars)[,2]
mpg        cyl       disp         hp       drat         wt       qsec         vs         am       gear       carb
-0.8521620  1.0000000  0.9020329  0.8324475 -0.6999381  0.7824958 -0.5912421 -0.8108118 -0.5226070 -0.4926866  0.5269883
> corrplot(cor(mtcars)) You can also reduce features by selecting them so that the maximum correlation is bounded (exclude the target variable from the process):

> toRemove <- findCorrelation(cor(mtcars[,-2]), cutoff = 0.8)
> corrplot(cor(cbind(mtcars[,-toRemove], cyl=mtcars[,2]))) 