Reproducible benchmarks for the performance of statistical prediction methods? There are many statistical models used for predictive modelling. These include famous methods such as naive Bayes, knn, SVM, random forest etc.
I am looking for reproducible examples (preferably in R) that demonstrates the performance of various prediction methods. Where can I find such resources?
Thanks.
 A: Tentative answer here but, well, there's a paper [1] comparing the performance of 22 classification algorithms predicting software failures in 10 public domain NASA Metrics Data repository datasets.
The data used in this study stems from the NASA MDP repository [10]. Ten software defect prediction data sets are analyzed, including the eight sets used in [44] as well as two additional data sets (JM1 and KC1, see also Table 1). Each data set is comprised of several software modules, together with their number of faults and characteristic code attributes.
The benchmarking experiment aims at contrasting the competitive performance of several classification algorithms. To that end, an overall number of 22 classifiers is selected, which may be grouped into the categories of statistical approaches, nearest-neighbor methods, neural networks, support vector machines, tree-based methods, and ensembles. The selection aims at achieving a balance between established techniques, such as Naive Bayes, decision trees, or logistic regression, and novel approaches that have not yet found widespread usage in defect prediction (e.g., different variants of support vector machines, logistic model trees, or random forests). The classifiers are sketched in Table 2, together with a brief description of their underlying paradigms.
I also found another paper [2] comparing some algorithms in two ecological modelling datasets. It has interesting discussion of the ease of use of such algorithms and tries to predict the probability of finding specimen given geographic features (check Fig. 3 and Fig. 4).
Logistic Multiple Regression, Principal Component Regression and Classification and Regression Tree Analysis (CART), commonly used in ecological modelling using GIS, are compared with a relatively new statistical technique, Multivariate Adaptive Regression Splines (MARS), to test their accuracy, reliability, implementation within GIS and ease of use. All were applied to the same two data sets, covering a wide range of conditions common in predictive modelling, namely geographical range, scale, nature of the predictors and sampling method.
The Grimmia data set (1285 cases; 419 present, 866 absent) represents the distribution of species of the moss genus Grimmia in Latin America, from Mexico to Cape Hornos (Fig. 3). Grimmia was recently revised for Latin America and its taxonomy is well known worldwide (Muñoz 1999; Muñoz & Pando 2000).
The Fagus data set (103 181 cases; ca. 50% each of presences and absences) was selected to represent high spatial resolution at a regional scale. The dependent variable was the presence/absence of Fagus sylvatica oligotrophic forest in the La Liébana region (Cantabria Province, NW Spain).
[1] Lessmann, S., Baesens, B., Mues, C., & Pietsch, S. (2008). Benchmarking classification models for software defect prediction: A proposed framework and novel findings. Software Engineering, IEEE Transactions on, 34(4), 485-496. [PDF at IEEE]
[2] Muñoz, J., & Felicísimo, Á. M. (2004). Comparison of statistical methods commonly used in predictive modelling. Journal of Vegetation Science, 15(2), 285-292.[PDF at Wiley]
