6
$\begingroup$

There are many statistical models used for predictive modelling. These include famous methods such as naive Bayes, knn, SVM, random forest etc.

I am looking for reproducible examples (preferably in R) that demonstrates the performance of various prediction methods. Where can I find such resources?

Thanks.

$\endgroup$
  • 3
    $\begingroup$ This is very general as it stands -- for us to compare relative performance we'd have to at least make some assumptions or restrictions to the kind of application/characteristics we're dealing with. The no free lunch theorem(s) would suggest that without some such identified restriction there's not really any advantage. Anyone who does come to some conclusion will therefore be talking about some particular domain of problems, and then their conclusions may not necessarily apply to somewhat different domains. $\endgroup$ – Glen_b -Reinstate Monica Apr 22 '16 at 6:40
  • $\begingroup$ Hi @Glen_b - I accept you comment and modified my question. Is the updated question better phrased? (thanks) $\endgroup$ – Tal Galili Apr 23 '16 at 7:57
  • 1
    $\begingroup$ OK, sorry, I misunderstood what you wanted. $\endgroup$ – usεr11852 Apr 26 '16 at 8:19
  • 1
    $\begingroup$ Maybe it'd help if you could describe what the ideal answer to your question would look like. Does that make sense? Also, is there a reason why you don't just take some of the more famous data sets from the ML repository and make the models yourself? If you use a package like caret then the whole process would be pretty easy. $\endgroup$ – Steve S Apr 28 '16 at 21:17
  • 1
    $\begingroup$ scikit-learn.org/stable/auto_examples/classification/… ? $\endgroup$ – shadowtalker Apr 30 '16 at 13:39
2
+50
$\begingroup$

Tentative answer here but, well, there's a paper [1] comparing the performance of 22 classification algorithms predicting software failures in 10 public domain NASA Metrics Data repository datasets.

The data used in this study stems from the NASA MDP repository [10]. Ten software defect prediction data sets are analyzed, including the eight sets used in [44] as well as two additional data sets (JM1 and KC1, see also Table 1). Each data set is comprised of several software modules, together with their number of faults and characteristic code attributes.
The benchmarking experiment aims at contrasting the competitive performance of several classification algorithms. To that end, an overall number of 22 classifiers is selected, which may be grouped into the categories of statistical approaches, nearest-neighbor methods, neural networks, support vector machines, tree-based methods, and ensembles. The selection aims at achieving a balance between established techniques, such as Naive Bayes, decision trees, or logistic regression, and novel approaches that have not yet found widespread usage in defect prediction (e.g., different variants of support vector machines, logistic model trees, or random forests). The classifiers are sketched in Table 2, together with a brief description of their underlying paradigms.

I also found another paper [2] comparing some algorithms in two ecological modelling datasets. It has interesting discussion of the ease of use of such algorithms and tries to predict the probability of finding specimen given geographic features (check Fig. 3 and Fig. 4).

Logistic Multiple Regression, Principal Component Regression and Classification and Regression Tree Analysis (CART), commonly used in ecological modelling using GIS, are compared with a relatively new statistical technique, Multivariate Adaptive Regression Splines (MARS), to test their accuracy, reliability, implementation within GIS and ease of use. All were applied to the same two data sets, covering a wide range of conditions common in predictive modelling, namely geographical range, scale, nature of the predictors and sampling method.
The Grimmia data set (1285 cases; 419 present, 866 absent) represents the distribution of species of the moss genus Grimmia in Latin America, from Mexico to Cape Hornos (Fig. 3). Grimmia was recently revised for Latin America and its taxonomy is well known worldwide (Muñoz 1999; Muñoz & Pando 2000).
The Fagus data set (103 181 cases; ca. 50% each of presences and absences) was selected to represent high spatial resolution at a regional scale. The dependent variable was the presence/absence of Fagus sylvatica oligotrophic forest in the La Liébana region (Cantabria Province, NW Spain).

[1] Lessmann, S., Baesens, B., Mues, C., & Pietsch, S. (2008). Benchmarking classification models for software defect prediction: A proposed framework and novel findings. Software Engineering, IEEE Transactions on, 34(4), 485-496. [PDF at IEEE]

[2] Muñoz, J., & Felicísimo, Á. M. (2004). Comparison of statistical methods commonly used in predictive modelling. Journal of Vegetation Science, 15(2), 285-292.[PDF at Wiley]

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for?Browse other questions tagged or ask your own question.