6
$\begingroup$

I'm learning about the Decision Tree and Random Forests. But there is something I don't really understand.

I have a training set and a cross-validation set. I need to train different Random Forests, each with a different number of trees. For each forest, I need to plot the classification score for the training set and the cross-validation set (a validation curve).

But what is the classification score for a Random Forest? Do I need to count the number of misclassifications? And how do I plot this?

PS: I use the Python SciKit Learn package

$\endgroup$
  • $\begingroup$ what do you use to run RF? In randomForest package in R there is plot.randomForest() which plots error rates or MSE of a randomForest object. $\endgroup$ – katya Nov 27 '14 at 19:44
  • $\begingroup$ I use the Python SciKit Learn package. $\endgroup$ – JNevens Nov 27 '14 at 19:46
10
$\begingroup$

You typically plot a confusion matrix of your test set (recall and precision), and report an F1 score on them.
If you have your correct labels of your test set in y_test and your predicted labels in pred, then your F1 score is:

from sklearn import metrics
# testing score
score = metrics.f1_score(y_test, pred, pos_label=list(set(y_test)))
# training score
score_train = metrics.f1_score(y_train, pred_train, pos_label=list(set(y_train)))

These are the scores you likely want to plot.

You can also use accuracy:

pscore = metrics.accuracy_score(y_test, pred)
pscore_train = metrics.accuracy_score(y_train, pred_train)

However, you get more insight from a confusion matrix.

You can plot a confusion matrix like so, assuming you have a full set of your labels in categories:

import numpy as np, pylab as pl
# get overall accuracy and F1 score to print at top of plot
pscore = metrics.accuracy_score(y_test, pred)
score = metrics.f1_score(y_test, pred, pos_label=list(set(y_test)))
# get size of the full label set
dur = len(categories)
print "Building testing confusion matrix..."
# initialize score matrices
trueScores = np.zeros(shape=(dur,dur))
predScores = np.zeros(shape=(dur,dur))
# populate totals
for i in xrange(len(y_test)-1):
  trueIdx = y_test[i]
  predIdx = pred[i]
  trueScores[trueIdx,trueIdx] += 1
  predScores[trueIdx,predIdx] += 1
# create %-based results
trueSums = np.sum(trueScores,axis=0)
conf = np.zeros(shape=predScores.shape)
for i in xrange(len(predScores)):
  for j in xrange(dur):
    conf[i,j] = predScores[i,j] / trueSums[i]
# plot the confusion matrix
hq = pl.figure(figsize=(15,15));
aq = hq.add_subplot(1,1,1)
aq.set_aspect(1)
res = aq.imshow(conf,cmap=pl.get_cmap('Greens'),interpolation='nearest',vmin=-0.05,vmax=1.)
width = len(conf)
height = len(conf[0])
done = []
# label each grid cell with the misclassification rates
for w in xrange(width):
  for h in xrange(height):
      pval = conf[w][h]
      c = 'k'
      rais = w
      if pval > 0.5: c = 'w'
      if pval > 0.001:
        if w == h:
          aq.annotate("{0:1.1f}%\n{1:1.0f}/{2:1.0f}".format(pval*100.,predScores[w][h],trueSums[w]), xy=(h, w), 
                  horizontalalignment='center',
                  verticalalignment='center',color=c,size=10)
        else:
          aq.annotate("{0:1.1f}%\n{1:1.0f}".format(pval*100.,predScores[w][h]), xy=(h, w), 
                  horizontalalignment='center',
                  verticalalignment='center',color=c,size=10)
# label the axes
pl.xticks(range(width), categories[:width],rotation=90,size=10)
pl.yticks(range(height), categories[:height],size=10)
# add a title with the F1 score and accuracy
aq.set_title(lbl + " Prediction, Test Set (f1: "+"{0:1.3f}".format(score)+', accuracy: '+'{0:2.1f}%'.format(100*pscore)+", " + str(len(y_test)) + " items)",fontname='Arial',size=10,color='k')
aq.set_ylabel("Actual",fontname='Arial',size=10,color='k')
aq.set_xlabel("Predicted",fontname='Arial',size=10,color='k')
pl.grid(b=True,axis='both')
# save it
pl.savefig("pred.conf.test.png")

and you end up with something like this (example from LiblinearSVC model), where you look for a darker green for better performance, and a solid diagonal for overall good performance. Labels missing from the test set show as empty rows. This also gives you a good visual of what labels are being misclassified as. For example, take a look at the "Music" column. You can see along the diagonal that 75.7% of the items that were predicted to be "Music" where actually "Music". Travel along the column and you can see what the other labels really were. There was clearly some confusion with music-related labels, like "Tuba", "Viola", "Violin", indicating that perhaps "Music" is too general to try and predict if we can be more specific.

Confusion Matrix example

$\endgroup$
  • $\begingroup$ What exactly is a Confusion Matrix? I don't really know what it is. Also, what is a Validation Curve? $\endgroup$ – JNevens Nov 28 '14 at 17:09
  • 2
    $\begingroup$ A confusion matrix shows you what labels are being confused with other labels. It's typically a square grid, like my example, with rows giving the true label and columns giving the predicted label from your RF model. "Validation Curve" was in your post, but I'm assuming you mean to construct a ROC curve for each label and look at the area under the curve -- see Wikipedia ROC Curve -- and then compare them across models. $\endgroup$ – wwwslinger Nov 28 '14 at 19:06
  • $\begingroup$ I think I can use the ROC curve more than the confusion matrix, because I have a binary classifier $\endgroup$ – JNevens Nov 28 '14 at 19:20
  • 3
    $\begingroup$ A confusion matrix with a binary classifier still works (you'll just have a 2x2 matrix), but area under the ROC is pretty standard, so I'd go with that. scikit-learn has a function for that as well: roc_curve $\endgroup$ – wwwslinger Nov 28 '14 at 22:26

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.