What key information should be included in an academic paper that uses machine learning? Imagine you are reviewing a manuscript that describes application of a supervised machine learning algorithm (e.g. SVM, CART, logistic regression, random forest etc.) to predict a binary output.  Perhaps they've even applied all of them to the same problem.  
What would be the optimal information that should be included in a report using classification algorithms?
Essentially, I'm asking this question (Reporting results of simple linear regression: what information to include?), but for classification algorithms.
This is what I've devised so far:

  
*
  
*Information about the data (n, sample, source, rates of each event class, # of predictors)
  
*If/how it was split into training and test (60/40, crossvalidation)
  
*Performance of the algorithm at classification (OOB, error rate, accuracy, efficiency, ROC AUC, TP/FP)
  
*Variable importance/significance (Gini, p-values, OR)
  
*Overfitting/lack of fit parameters (AIC, residuals, plots of fit, QQ plots)
  

While ignorant, I anticipate the answer will depend on the algorithm used, so please feel free to break it down by method (or tell me that this can't/shouldn't be generalized and why).
 A: I'll start off your list and both comment on it and add something.


*

*Information about the data (n, sample, source, rates of each event class, # of predictors)


*

*Possible biases in sampling the data, e.g., Most people are not WEIRD

*Descriptive plots for predictors: marginal and pairwise distributions, histogram/beanplots and scatterplots


*If/how it was split into training and test (60/40, crossvalidation) 


*

*If cross-validation: was this run once or multiple times?


*Model selection and training


*

*What models were available, and how were they selected (or combined, for ensemble methods)?

*How were they trained?


*Performance of the algorithm at classification (OOB, error rate, accuracy, efficiency, ROC AUC, TP/FP)


*

*Accuracy, error rate, precision, TPR, FPR etc. are not good quality measures. Better to use scoring-rules instead. Of course, these should be assessed out-of-sample.

*Always compare your new model to a simple benchmark


*Variable importance/significance (Gini, p-values, OR)


*

*Variable importance is important. Time spent reading up on variable importance measures is time well spent

*p-values do not measure variable importance

*For ORs, you first need to discretize continuous predictors, which is not a good idea


*Overfitting/lack of fit parameters (AIC, residuals, plots of fit, QQ plots)


*

*Overfitting is important. Best to report OOB scoring rules

*Many ML algorithms have no notion of AIC or other information criteria, because they don't have an underlying statistical model, or the model is almost unknown to ML researchers. Plus, ICs are not always comparable between different models

*Residuals and q-q plots make more sense in numerical prediction than in classification


