Why is my learning curve always at 100% while my validation curve increases with training samples?

Question

I am relatively new to machine learning and have the following problem:

I have built a random forest model which works relatively well and now I am trying to interpret the results.

The learning curve looks like this:

Now my question: How can it be that the training accuracy is always 1?

The code:

from sklearn.model_selection import learning_curve

train_sizes, train_scores, test_scores =\
    learning_curve(estimator = RandomForestClassifier(n_estimators=100), X = X_train, y = y_train, train_sizes = np.linspace(0.1,1,5), cv  = 5, n_jobs = -1)

train_mean = np.mean(train_scores, axis=1)
train_std = np.std(train_scores, axis=1)
test_mean = np.mean(test_scores, axis=1)
test_std = np.std(test_scores, axis=1)

plt.plot(train_sizes, train_mean,
         color = "blue", marker = 'o',
         markersize = 5,label  ='Training accuracy')

plt.fill_between(train_sizes, 
                 train_mean + train_std, 
                 train_mean - train_std,
                 alpha=0.15, color = 'blue')

plt.plot(train_sizes, test_mean, 
         color='red', linestyle = '--',
         marker = 's',markersize = 5, 
         label = 'Validation accuracy')

plt.fill_between(train_sizes, 
                 test_mean + test_std, 
                 test_mean - test_std,
                 alpha=0.15, color = 'green')
    

plt.grid()
plt.xlabel('Number of training examples')
plt.ylabel('Accuracy')
plt.legend(loc = 'lower right')
plt.ylim([0.25, 1.01])
plt.show

Thanks for help

Answer 1

How can it be that the training accuracy is always 1?

That's by construction of the decision trees in the RF: (at least by default) the trees are not pruned. Thus roughly 2/3 (more precisely, approximately 1 - 1/e) of the trees will contain the training case and can look up the correct solution. Even if the 1/e trees which were not trained with this case would always predict wrongly, they'd always be outvoted.

Conclusion: for RF, only oob error i.e. using only those trees that were not trained on the case or validation/vreification with proper unknown cases are relevant.