6
$\begingroup$

I ran the following code in R:

rf.classifier.master <- randomForest(my_response ~ ., data=feature.matrix)
print(rf.classifier.master)

and got the following results:

enter image description here

-1 corresponds to regular and 200 corresponds to special. However, I'd really like a less aggressive classifier, ideally with a maximum false positive rate (predict a 200 when they are actually a -1) of 5% or 7%, even if this means the overall error rate increases.

Is there a way to achieve this in R?

EDIT

I'm also really interested if there is a way to train two random forests with the same data, one where the false positive rate is ~ 0 and one where all true 200 observations are correctly classified as 200 (whats the term for this again, precision?), where overall error rate is maximized after those two conditions are met.

$\endgroup$
1
  • $\begingroup$ Also, if there is another competitive algorithm that can do this type of thing easier, i.e SVM, i'd also be interested in that. $\endgroup$ – mt88 Jul 29 '15 at 21:35
7
$\begingroup$

The default RF classification aggregate trees by majority vote. Either you must modify the distribution of class votes of trees(see example A) or you must change the aggregation rule (see example B). Option A could be achieved by stratification/downsampling or classweight. I mainly mention because it is possible, as it probably will decrease overall prediction performance (AUC of ROC of test set predictions). Option B is to modify aggregation rule. Any sample predicted by a forest will get a number of votes(or 0) on each of the classes. The pluralism of the votes can be understood as a pseudo estimate of predicted probability, where the predicted probability of k'th class is votes on class k divided by all-votes. The voting threshold can be modified with the cutoff parameter, either during training or during prediction. The predicted class probabilities are basically divided with the class cutoffs. if cutoff = c(.5,.5) there is no change. if cutoff = c(.1,.9) much more votes on class 1. There is a gotcha in the randomForest, such that OOB-CV predictions only will take effect from cutoff if modified during training, whereas for predictions of newdata or testsets cutoff can be modified after training.

library(randomForest)
make.data = function(obs=1000,vars=6) {
  X = data.frame(replicate(vars,rnorm(obs)))
  noise=rnorm(obs)
  y.value = with(X,X1^2+sin(X2)+X3*X4) + noise
  y.class = factor(y.value>median(y.value),labels=c("-1","200"))
  return(data.frame(y=y.class,X=X))
}
train.data = make.data() 
test.data = make.data()

#native RF
RF.default = randomForest(y~.,data=train.data)
print(RF.default)

>Confusion matrix:
>          predClass
>trainClass  -1 200
>       -1  386 114 (~22% false positive class 200)
>       200 131 369

#solution A: Unbalancing the data by stratification.
#It works, but not recommendable.
#stratified RF, downsample, false postive class "200" is  ~5.2%
RF.stratify = randomForest(y~.,data=train.data,
                          sampsize=c(500,140),
                          strata=train.data$y)
print(RF.stratify)

>Confusion matrix:
>     -1 200 class.error
>-1  468  32       0.064 (~10% false positive class 200)
>200 236 264       0.472

#solution B:
#changed vote-rule with cutoff
RF.default$forest$cutoff=c(.17,.83)
#cutoff is not implemented for OOB-CV in predict.randomForest!
preds.train = predict(RF.default)
table(trainClass=train.data$y,
       predClass=preds.train)
>          predClass
>trainClass  -1 200
>       -1  389 111 (OOB take no effect from cutoff after training)
>       200 108 392



#but it does work for newdata prediction
preds.test = predict(RF.default,newdata=test.data)
table(testClass=test.data$y,
      predClass=preds.test)

>         predClass
>testClass  -1 200
>      -1  487  13 (~10% false positive class 200)
>      200 362 138

#found the 'gotcha' in the source file of randomForest
#cutoff only modifies OOB predictions if modified during training
RF.default = randomForest(y~.,data=train.data,cutoff=c(.17,.83))
preds.train = predict(RF.default)
table(trainClass=train.data$y,
      predClass=preds.train)

>         predClass
>trainClass  -1 200
>       -1  490  10
>       200 366 134 


#extra tip use a ROC plot to investigate the relationship between false positive and false negative, to helpt choose your favorite cutoff.
library(AUC)
plot(roc(predict(RF.default,type="vote")[,2],train.data$y))
$\endgroup$
1
  • 2
    $\begingroup$ Thanks. I went with option B. It was a really easy fix, just had to do a ROC curve and have my random forest output probabilities. $\endgroup$ – mt88 Jul 30 '15 at 0:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.