I have a multiclass classification problem, for which I wanted to try different methods.
I tried multinomial logistic regression, random forests and XGBoost. I evaluated the methods with the same metrics, but my main ones are Brier score and Log loss.
I was surprised to see that the regression, even if it got larger Brier and log loss, the difference in performance between the train and test set is almost null (Brier 0.21 train, 0.22 test; Log loss 0.40 train, 0.42 test).
In the random forests and xgboost case, the opposite happens: the train scores are very very low (around 0.0 brier and 0.06 log loss) while the test scores are quite different (0.15 brier, 0.29 log loss).
As far as I'm concerned, it should be hard to overfit in random forests. Besides, this was done in 10 fold cross validation and a lot of things were taken into account to avoid overfitting. Also, someone else told me in a previous question of mine that having a different train/test score does not necessarily mean it is overfitting, and that, in any case, it does not mean the model is crap per se.
Does anyone have an idea of what could be happening? If you had any reference, I would be very grateful since this is for a paper.