5
$\begingroup$

TL;DR: I am working with binary classifications. I have different models I want to compare their performance out of the box. I read that accuracy is a poor metric, and Brier score or log loss should be used instead. However, I also read that the Brier score should not be used when comparing logistic regression vs. random forest, and it should be mainly used as a metric when tuning/changing the parameters of a single model. Is this statement true? Is it wrong to use Brier to compare the performance of different models/approaches?

Full background to my research question:

Hi all,

I have a dataset composed of two groups (disease type 1 vs. type 2) and 50 samples per group. For each sample, I have around 7000 features being measured. Importantly, identifying type 2 is key, and I am willing to "pay the price" of getting some type 1 as false positives.

My initial plan was to run feature selection and machine learning to classify these groups. After reading a bunch of stuff here, I realize that my approach may not be ideal for my dataset. For instance, ML with 100 samples is far from ideal. In addition, my dataset is 50/50 while the real-world prevalence of both disease types is 70/30; thus, any model I come up with will most likely underperform in the future.

I am aware of these limitations (and there are probably many more), but since the data is already in my hands right now, I wish I could "play" with it to see what I can get. I plan to run repeated k-fold cross-validation (10-fold with 10 repetitions). Inside each fold, I am performing mRMR (feature selection) and a few classification models. For example, logistic regression, random forest, SVM, XGBoost, and a few more. I want to compare the performance of each model and then spend more time optimizing the one that performed the best out of the box.

At first, I was going to compare log reg and the ML models using accuracy, but great posts by Frank Harrell, Stephan Kolassa, and others are changing my mind. Right now, I am planning to use Brier Score, at least in this initial stage where an overall screening is needed. However, I read that the Brier Score should not be used to compare logistic regression vs. random forest, as they are two different models. It seemed like Brier score should be used only for the same model under different parameters, for example, when evaluating the gains for hyperparameter tuning. How much of that is actually true?

Improve this question
$\endgroup$
18
  • 5
    $\begingroup$ Where did you read not to compare the Brier scores of different models? That makes no sense to me. Could that author have meant for in-sample assessments? // It might be best to edit that "log reg" to specify "logistic regression" instead of a regression on the logarithm of a value or a GLM with a logarithm link function. $\endgroup$
    – Dave
    Mar 29 at 19:19
  • 5
    $\begingroup$ People compare different models using proper scoring rules all the time, whether for classification or for numerical prediction. I have never hear that this should not be done, and that the models differ is not an argument - in a forecasting situation, you can absolutely compare ARIMA and RNNs using the MSE or the CRPS. So with Dave, I would be interested where you read this particular piece of advice, $\endgroup$
    – Stephan Kolassa
    Mar 29 at 19:37
  • 1
    $\begingroup$ @StephanKolassa I wonder if the suggestion comes from using a predict method on the random forest that return the category with the highest probability, and then if you calculate the MSE between a vector of numbers that represent true categories and another vector of numbers that represents the predicted categories, the calculation is not the Brier score (and not really a mean squared error, either). $\endgroup$
    – Dave
    Mar 29 at 20:05
  • 2
    $\begingroup$ Of course, there was no reference providing the basis for his statement, so I went on to the literature to see if I could find anything close to this, and I found this publication: The Brier score does not evaluate the clinical utility of diagnostic tests or prediction models (Assel et al; PMID: 31093548). These two pieces of information confused me a little about how and if I should apply Brier scoring. $\endgroup$
    – Luiz Gustavo
    Mar 29 at 23:34
  • 1
    $\begingroup$ I looked at the Assel et al. article, and it says what @Dave agrees on and where I concur: the Brier score alone will not tell you whether a classification-decision pipeline is useful. It will only evaluate the classification model as to its calibration. Whether the entire pipeline is more useful than another one needs to be seen in its context. $\endgroup$
    – Stephan Kolassa
    Mar 31 at 14:13

1 Answer 1

Reset to default
1
$\begingroup$

Brier score might not be the statistic of interest for a particular task. In that case, Brier score would not be appropriate for comparing a logistic regression and a random forest, but that is because Brier score simply is not the right value to calculate, rather than anything specific to how the values the Brier score evaluates are calculated or estimated.

However, if Brier score is what interests you, do calculate it. As long as the inputs to the score are appropriate (have an interpretation as probabilities, so not the log-odds output of a logistic regression or the predicted category that you can get by prediction methods in random forest software), go for it.

If there is an objection to doing this because random forests often give probability values that lack calibration and the Brier score will penalize this, that seems like a feature, not a bug, of Brier score. (Or maybe you don’t care about calibration, but then Brier score should not be your statistic of interest.)

If there is an objection to calculating the Brier score of a model because the model was not optimized well (mentioned in the comments), that seems like an admission that the model is not very good. If a model is making poor predictions (in terms of Brier score) because it was not optimized well, the key part of that to me is that the model is making poor predictions.

Improve this answer
$\endgroup$
3
  • $\begingroup$ In summary, if I have N methods classifying the same dataset, the Brier score is a valid way to compare their performance. This is a purely probabilistic approach and free of any "bias." Maybe a specific threshold for decision would give slightly different results when evaluating performance compared to the Brier, but that needs to be considered on a case-by-case basis. Did I understand it correctly? $\endgroup$
    – Luiz Gustavo
    Apr 3 at 20:43
  • $\begingroup$ @LuizGustavo I do not see where any kind of threshold comes up in my answer. Could you please explain what you mean? $\endgroup$
    – Dave
    Apr 3 at 20:44
  • $\begingroup$ Hi @Dave. I am so sorry for the late reply, life has been quite hectic lately. As for the threshold, I actually took this from one of Stephan's comment, stating that "Brier score is not the end-all if the classification model is used in a specific decision context - hard 0-1 classifiers with specific sensitivities and specificities may yield better decisions". This is why I assumed specific thresholds could still be useful in specific scenarios. $\endgroup$
    – Luiz Gustavo
    Apr 19 at 21:26

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.