4
$\begingroup$

Imagine you are reviewing a manuscript that describes application of a supervised machine learning algorithm (e.g. SVM, CART, logistic regression, random forest etc.) to predict a binary output. Perhaps they've even applied all of them to the same problem.

What would be the optimal information that should be included in a report using classification algorithms?

Essentially, I'm asking this question (Reporting results of simple linear regression: what information to include?), but for classification algorithms.

This is what I've devised so far:

  • Information about the data (n, sample, source, rates of each event class, # of predictors)
  • If/how it was split into training and test (60/40, crossvalidation)
  • Performance of the algorithm at classification (OOB, error rate, accuracy, efficiency, ROC AUC, TP/FP)
  • Variable importance/significance (Gini, p-values, OR)
  • Overfitting/lack of fit parameters (AIC, residuals, plots of fit, QQ plots)

While ignorant, I anticipate the answer will depend on the algorithm used, so please feel free to break it down by method (or tell me that this can't/shouldn't be generalized and why).

Improve this question
$\endgroup$
4
  • 1
    $\begingroup$ It would depend on the practical use (skewed data? FP/FN costs? selectivity/sensitivity? inference? exploratory?). The choices of performance measures should arguable reflect that. I guess many of the mentioned measures could be a part of such selection. (personally would not use Gini and p-values). If more methods are to be compared, then models should be wrapped in the same outer cross validation such as 20-rep 10-fold CV or similar. $\endgroup$
    – Soren Havelund Welling
    Commented Dec 1, 2015 at 15:12
  • $\begingroup$ Would it be useful if I provided a detailed hypothetical example to establish a framework for key information? Not trying to make work, but provide a starting point for neophytes. $\endgroup$
    – Minnow
    Commented Dec 1, 2015 at 15:14
  • $\begingroup$ Yes that would help. Here's some suggentions on what to include: meta.stats.stackexchange.com/questions/1479/… $\endgroup$
    – Soren Havelund Welling
    Commented Dec 1, 2015 at 15:19
  • $\begingroup$ It needs to be reproducible, so there needs to be enough information to derive the same result from the data from the description given in the paper (the hyper-parameter tuning is the bit that many leave out, but unfortunately is crucial) $\endgroup$
    – Dikran Marsupial
    Commented Jan 18, 2022 at 15:21

1 Answer 1

Reset to default
4
$\begingroup$

I'll start off your list and both comment on it and add something.

  • Information about the data (n, sample, source, rates of each event class, # of predictors)
    • Possible biases in sampling the data, e.g., Most people are not WEIRD
    • Descriptive plots for predictors: marginal and pairwise distributions, histogram/beanplots and scatterplots
  • If/how it was split into training and test (60/40, crossvalidation)
    • If cross-validation: was this run once or multiple times?
  • Model selection and training
    • What models were available, and how were they selected (or combined, for ensemble methods)?
    • How were they trained?
  • Performance of the algorithm at classification (OOB, error rate, accuracy, efficiency, ROC AUC, TP/FP)
  • Variable importance/significance (Gini, p-values, OR)
    • Variable is important. Time spent reading up on variable importance measures is time well spent
    • p-values do not measure variable importance
    • For ORs, you first need to discretize continuous predictors, which is not a good idea
  • Overfitting/lack of fit parameters (AIC, residuals, plots of fit, QQ plots)
    • Overfitting is important. Best to report OOB scoring rules
    • Many ML algorithms have no notion of AIC or other information criteria, because they don't have an underlying statistical model, or the model is almost unknown to ML researchers. Plus, ICs are not always comparable between different models
    • Residuals and q-q plots make more sense in numerical prediction than in classification
Improve this answer
$\endgroup$
4
  • 3
    $\begingroup$ I would add that one should report what software (what programming language, what library, what package) was use to fit the model. $\endgroup$
    – amoeba
    Commented Nov 22, 2017 at 12:31
  • $\begingroup$ +1 (especially model selection bit) I disagree about accuracy. In some applications, it is the quantity of interest in the practical application (e.g. stats.stackexchange.com/questions/312780/…), and where that is the case it needs to be included in the performance evaluation. Accuracy is probably not going to be a good model selection criterion and proper scoring rules are better there, but it is important to bear in mind that model selection and performance evaluation are not the same thing. $\endgroup$
    – Dikran Marsupial
    Commented Jan 18, 2022 at 15:25
  • $\begingroup$ I would include how any hyper-parameters were tuned in the model selection and training bit. ML is particularly bad about that, especially as the hyper-parameters are very important for many ML methods. $\endgroup$
    – Dikran Marsupial
    Commented Jan 18, 2022 at 15:28
  • $\begingroup$ Also ML has alternatives to AIC, such as VC dimension, bounds on generalisation performance, MDL etc. ML tends to be more in favour of out-of-sample evaluation. I am not altogether sure why statisticians would prefer AIC to thorough out of sample evaluation, at least in applications with sufficient data available. $\endgroup$
    – Dikran Marsupial
    Commented Jan 18, 2022 at 15:34

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.