4
$\begingroup$

Imagine you are reviewing a manuscript that describes application of a supervised machine learning algorithm (e.g. SVM, CART, logistic regression, random forest etc.) to predict a binary output. Perhaps they've even applied all of them to the same problem.

What would be the optimal information that should be included in a report using classification algorithms?

Essentially, I'm asking this question (Reporting results of simple linear regression: what information to include?), but for classification algorithms.

This is what I've devised so far:

  • Information about the data (n, sample, source, rates of each event class, # of predictors)
  • If/how it was split into training and test (60/40, crossvalidation)
  • Performance of the algorithm at classification (OOB, error rate, accuracy, efficiency, ROC AUC, TP/FP)
  • Variable importance/significance (Gini, p-values, OR)
  • Overfitting/lack of fit parameters (AIC, residuals, plots of fit, QQ plots)

While ignorant, I anticipate the answer will depend on the algorithm used, so please feel free to break it down by method (or tell me that this can't/shouldn't be generalized and why).

$\endgroup$
  • 1
    $\begingroup$ It would depend on the practical use (skewed data? FP/FN costs? selectivity/sensitivity? inference? exploratory?). The choices of performance measures should arguable reflect that. I guess many of the mentioned measures could be a part of such selection. (personally would not use Gini and p-values). If more methods are to be compared, then models should be wrapped in the same outer cross validation such as 20-rep 10-fold CV or similar. $\endgroup$ – Soren Havelund Welling Dec 1 '15 at 15:12
  • $\begingroup$ Would it be useful if I provided a detailed hypothetical example to establish a framework for key information? Not trying to make work, but provide a starting point for neophytes. $\endgroup$ – Minnow Dec 1 '15 at 15:14
  • $\begingroup$ Yes that would help. Here's some suggentions on what to include: meta.stats.stackexchange.com/questions/1479/… $\endgroup$ – Soren Havelund Welling Dec 1 '15 at 15:19
2
$\begingroup$

I'll start off your list and both comment on it and add something.

  • Information about the data (n, sample, source, rates of each event class, # of predictors)
    • Possible biases in sampling the data, e.g., Most people are not WEIRD
    • Descriptive plots for predictors: marginal and pairwise distributions, histogram/beanplots and scatterplots
  • If/how it was split into training and test (60/40, crossvalidation)
    • If cross-validation: was this run once or multiple times?
  • Model selection and training
    • What models were available, and how were they selected (or combined, for ensemble methods)?
    • How were they trained?
  • Performance of the algorithm at classification (OOB, error rate, accuracy, efficiency, ROC AUC, TP/FP)
  • Variable importance/significance (Gini, p-values, OR)
    • Variable is important. Time spent reading up on variable importance measures is time well spent
    • p-values do not measure variable importance
    • For ORs, you first need to discretize continuous predictors, which is not a good idea
  • Overfitting/lack of fit parameters (AIC, residuals, plots of fit, QQ plots)
    • Overfitting is important. Best to report OOB scoring rules
    • Many ML algorithms have no notion of AIC or other information criteria, because they don't have an underlying statistical model, or the model is almost unknown to ML researchers. Plus, ICs are not always comparable between different models
    • Residuals and q-q plots make more sense in numerical prediction than in classification
$\endgroup$
  • 2
    $\begingroup$ I would add that one should report what software (what programming language, what library, what package) was use to fit the model. $\endgroup$ – amoeba says Reinstate Monica Nov 22 '17 at 12:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.