1
$\begingroup$

In my training dataset I have two classes. I have run rf using the following code

control <- trainControl(method = "repeatedcv", repeats=5, p=.7, sampling = "down") 
fit  <- train(x = training, y = factor(classe), method = "rf", trControl = control, importance=TRUE)

#> fit
#    Random Forest
#    
#     27 samples
#    403 predictors
#      2 classes: 'group1', 'group2'

But rf variable importance returns one overal value

#rf variable importance
#
#  only 20 most important variables shown (out of 403)
#
#             Overal
#v1           100.00
#v6           96.03
#v8           94.74
#v33          94.18
#v34          92.22
#v67          91.81

and trying

head(varImp(fit)$importance)
#           group1 group2
#v13        20.87988 20.87988
#v45        20.87988 20.87988
#v67        20.87988 20.87988
#v56        20.87988 20.87988
#v3         20.87988 20.87988
#v34        92.22880 92.22880

Returns exactly the same values while I want to have importance across classes. Is there any option that I should add it in the command?

$\endgroup$

closed as off-topic by Xi'an, Michael Chernick, John, Peter Flom Jan 5 '17 at 13:22

This question appears to be off-topic. The users who voted to close gave this specific reason:

  • "This question appears to be off-topic because EITHER it is not about statistics, machine learning, data analysis, data mining, or data visualization, OR it focuses on programming, debugging, or performing routine operations within a statistical computing platform. If the latter, you could try the support links we maintain." – Xi'an, John, Peter Flom
If this question can be reworded to fit the rules in the help center, please edit the question.

  • $\begingroup$ Can you confirm every importance ended being the same? Otherwise I guess it might be only in caret, because using importance.randomForest results in different values. $\endgroup$ – Firebug Jan 5 '17 at 2:20
  • $\begingroup$ all the same... $\endgroup$ – user6013305 Jan 5 '17 at 2:22
3
$\begingroup$

The reason that the variable importance is overall and not per class is a result of the way that importance is calculated. The importance assigned to a variable is the mean decrease in impurity averaged over all nodes where that variable was used to split the node. Certainly, the nodes are impure before splitting and often afterwards, so there would be no clear class to assign this change to. A quick example may help. Suppose that you have a node that has 40 points - 10 of each of 4 classes. You use a variable to split the node into two children. Each child has 20 points - ten from each of two classes. This would be a pretty useful split, but there is no way to assign credit to just one of the classes. The change in impurity is a statement about how the whole population was subdivided. This "example" was about one node. The measure is just the average of these changes over all nodes using the given attribute. But the change in impurity is a statement about how well the feature did at simplifying the population, not about how well it worked to identify a particular class.

I think that you are also asking if there is a way to get a per-class importance. If you need such a measure, you could (with some work) get a measure of the importance of a variable in identifying a particular class by recoding the class variable into a binary variable. Given a specific value of the class to be predicted, class=A, you could create the binary variable classA which is 1 if class=A and 0 otherwise. Then compute the variable importance in predicting this feature. That should be useful in tying variables to a specific class. I doubt that there could be any useful comparisons of these measures across the different values of the class, that is, I don't think that you can compare the importance of a variable based on classA with the importance base on classB, but still for each class it would tell you something.

$\endgroup$
  • $\begingroup$ Thanks for you reply. how it can be solved? $\endgroup$ – user6013305 Jan 5 '17 at 9:12
  • $\begingroup$ Adding a little to the answer. $\endgroup$ – G5W Jan 5 '17 at 11:22
0
$\begingroup$
load("Data.rda")

set.seed(123)

trainindex<- createDataPartition(Data$Class , p=0.8, list=F)

Train<-Data[trainindex,]

Test<-Data[-trainindex,]

ctrl <- trainControl(method = "cv",   

                     number = 10, summaryFunction = twoClassSummary, 

                     classProbs = T )


grid_rf<- expand.grid(.mtry = c(1, 2, 7, 8, 12, 15))

set.seed(123)

m_rf <- train(Class ~ ., data = Train, method = "rf",   

              metric = "ROC", trControl = ctrl,         

              tuneGrid = grid_rf,importance=T)

### in case of ROC

### in case of accuracy

ctrl <- trainControl(method = "repeatedcv",   

                     number = 10, repeats = 5) 

grid_rf <- expand.grid(.mtry = c(1, 2, 7, 8, 12, 15))

set.seed(123)

m_rf <- train(Class~ ., data = Train, method = "rf",   

              metric = "Accuracy", trControl = ctrl,         

              tuneGrid = grid_rf,importance=T)

####you may also write kappa instead of Accuracy

## in this fashion everything must be working (give it a try); hope this helps
$\endgroup$
  • $\begingroup$ thanks. I just ran it again with exactly the same as yours except tuneGrid = grid_rf, is it important? $\endgroup$ – user6013305 Jan 5 '17 at 2:32
  • $\begingroup$ @user6013305 it is if you want to specify the mtry-s that you want your model be corss validated ( by default rf takes the square root of the number of variables say 10 out of 100 but you may want to check for exactly 5, 6, 7,8, 80. $\endgroup$ – Erik Hambardzumyan Jan 5 '17 at 2:34
  • $\begingroup$ @user6013305 it depends on the number of variables you have $\endgroup$ – Erik Hambardzumyan Jan 5 '17 at 2:35
  • $\begingroup$ @user6013305 have you tried varImp ? $\endgroup$ – Erik Hambardzumyan Jan 5 '17 at 2:38
  • $\begingroup$ the same. two classes have exactly the same importance value: varImp(fit)$importance $\endgroup$ – user6013305 Jan 5 '17 at 2:56

Not the answer you're looking for? Browse other questions tagged or ask your own question.