Benchmarking new techniques for Data Mining

Question

In The Elements of Statistical Learning (ESL), a benchmark of “Off-the-Shelf” Procedures for Data Mining is provided p371. They provide a benchmark comparing Neural Nets, SVM, Trees, MARS, KNN with kernels, in terms of handling of mixed data types, missing values, robustness (to outliers, to irrelevant inputs, to monotone transformation of inputs), ability to extract linear dependence, computational scalability, interpretability and predictive power. Another model benchmark is given by Top 10 algorithms in data mining, which provide an introduction to: C4.5, k-Means, SVM, Apriori, EM, PageRank, AdaBoost, kNN, Naive Bayes, and CART.

Those benchmarks date respectively from 2008 and 2007, while new techniques appeared since then. I am mostly thinking about recent papers : Residual Networks (2015), Capsules (2017), Neural Ordinary Differential Equations (2018).

Are there up to date benchmarks for more recent “Off-the-Shelf” procedures for Data Mining ? Or, alternatively, can we consider those models as subparts of Neural networks ? Not recent enough to be considered "Off the shelf", thus benchmarked ?

Answer 1

It is very difficult to compare algorithms in a very general manner. Specific situations may favor one algorithm over another, strong higher order interactions may favor tree based algorithms or neural networks over models that require us to specify these, very large datasets may favor neural networks/complex trees, having the right prior information may favor a Bayesian approach, image data (and various other types of data such as natural language) may well be the natural domain of neural networks today, in audio data a cleverly crafted hidden Markov models may beat neural networks (esp. with a limited amount of data), various small choices (e.g. data augmentation, good pre-processing affects different models differently) small datasets with features in an already well transformed form may heavily favor simple models such as traditional regression type models and so on.

There's also other considerations such as explainability/interpretability, runtime (see e.g. xgboost vs. LightGBM), resource needs (memory which can be an issue with various random forest implementations, need for GPUs and so on) and what you are trying to optimize. On the last point, you may favor extremely complex models, if you care about every last decimal point of performance and are sure that what you want to predict for comes from the same data generating mechanism as your training data (this is often the case on a kaggle competition), while you may favor a simpler model, where you need to collect less data in the future and understand deviations from the data generating mechanism better for a real-life situation.

A recent paper concluded that existing benchmarks are not diverse enough to truly benchmark methods - although there are of course useful sets of standard benchmarks for which people are developing tailored neural networks, which is perhaps for very specific tasks the main benchmark for NN approach that I can think of. I assume the question was asked in a more general sense, but I suspect there simply is no very general answer that some approach always works best. Another interesting recent publication looked at whether logistic regression was competitive with more modern ML methods in practical applications where different approaches were tried. On average there was not much of a difference, for which there could of course be many explanations (amongst them that LR is often a good approach, that not all approaches were tuned equally well and so on).

Answer 2

Hm, benchmarking neural networks is tricky for many reasons

• a) they are typically used on datasets where you would not apply traditional methods because the latter (off-the-shelf methods) would not perform well there (i.e., you would apply CNNs directly to images instead of some sort of landmark features extracted from images; you would apply RNNs to raw text data -- or word embeddings at the least -- instead of bag-of-word models)
• b) there are many, many more architectural settings and tuning parameters to consider

A recent study that comes to mind, although particular to a dataset, which I wouldn't consider ideal for DL -- and they only used a simple MLP model -- would be

• Koutsoukas A, Monaghan KJ, Li X, Huan J: Deep-learning: investigating deep neural networks hyper-parameters and comparison of performance to shallow methods for modeling bioactivity data. Journal of Cheminformatics 2017, 9.

For some benchmarks regarding CNN architectures, for example, consider

• Canziani, Alfredo, Adam Paszke, and Eugenio Culurciello. "An analysis of deep neural network models for practical applications." arXiv preprint arXiv:1605.07678 (2016).