Imagine you are reviewing a manuscript that describes application of a supervised machine learning algorithm (e.g. SVM, CART, logistic regression, random forest etc.) to predict a binary output. Perhaps they've even applied all of them to the same problem.
What would be the optimal information that should be included in a report using classification algorithms?
Essentially, I'm asking this question (Reporting results of simple linear regression: what information to include?), but for classification algorithms.
This is what I've devised so far:
- Information about the data (n, sample, source, rates of each event class, # of predictors)
- If/how it was split into training and test (60/40, crossvalidation)
- Performance of the algorithm at classification (OOB, error rate, accuracy, efficiency, ROC AUC, TP/FP)
- Variable importance/significance (Gini, p-values, OR)
- Overfitting/lack of fit parameters (AIC, residuals, plots of fit, QQ plots)
While ignorant, I anticipate the answer will depend on the algorithm used, so please feel free to break it down by method (or tell me that this can't/shouldn't be generalized and why).