16
$\begingroup$

The structure of this question is as follows: at first, I provide the concept of ensemble learning, further I provide a list of pattern recognition tasks, then I give examples of ensemble learning algorithms and, finally, introduce my question. Those who don't need all the supplemental information might just look at the headlines and go straight to my question.

What is ensemble learning?

According to Wikipedia article:

In statistics and machine learning, ensemble methods use multiple learning algorithms to obtain better predictive performance than could be obtained from any of the constituent learning algorithms alone. Unlike a statistical ensemble in statistical mechanics, which is usually infinite, a machine learning ensemble refers only to a concrete finite set of alternative models, but typically allows for much more flexible structure to exist among those alternatives.

Examples of pattern recognition tasks:

Examples of ensemble learning algorithms:

The following ensemble learning algorithms used for PR tasks (according to Wiki):

Ensemble learning algorithms (supervised meta-algorithms for combining multiple learning algorithms together):

  • Boosting (a machine learning ensemble meta-algorithm for primarily reducing bias, and also variance in supervised learning, and a family of machine learning algorithms which convert weak learners to strong ones)

  • Bootstrap aggregating ("bagging") (a machine learning ensemble meta-algorithm designed to improve the stability and accuracy of machine learning algorithms used in statistical classification and regression).

  • Ensemble averaging (the process of creating multiple models and combining them to produce a desired output, as opposed to creating just one model. Frequently an ensemble of models performs better than any individual model, because the various errors of the models "average out.")

  • Mixture of experts, hierarchical mixture of experts

Different implementations

  • Ensembles of neural networks (a set of neural network models taking a decision by averaging the results of individual models).
  • Random forest (an ensemble learning method for classification, regression and other tasks, that operate by constructing a multitude of decision trees at training time and outputting the class that is the mode of the classes (classification) or mean prediction (regression) of the individual trees).
  • AdaBoost (the output of the other learning algorithms ('weak learners') is combined into a weighted sum that represents the final output of the boosted classifier).

Additionally:

  • Methods that use one neural network to combine different classifiers
  • Competence areas method

My question

Which of the ensemble learning algorithms is considered to be state-of-the-art nowadays and is actually used in practice (for face detection, vehicle registration plates recognition, optical character recognition, etc.) by enterprises and organizations? Using ensemble learning algorithms is supposed to increase recognition accuracy and lead to a better computational efficiency. But, do matters stand this way in reality?

Which ensemble method, potentially, can show better classification accuracy and performance in the pattern recognition tasks? Perhaps, some of the methods are outdated now, or have shown to be ineffective. It is also possible that ensemble methods now tend to not be used anymore on the strength of some new algorithms. Those who have experience in the area or have sufficient knowledge in this field, can you help to clarify the matters?

Improve this question
$\endgroup$
3
  • $\begingroup$ What I heard recently is that people love XGBoost and it showed really impressive performance on several Kaggle competitions. $\endgroup$
    – Sangwoong Yoon
    Sep 3, 2016 at 14:15
  • $\begingroup$ The answer is short: the one that gives the best CV score. Usually it's stacking $\endgroup$
    – Alexey Grigorev
    Sep 4, 2016 at 16:31
  • $\begingroup$ The success and failure of an ensemble model is a function of the member models of the ensemble and the nature of the data. Ensemble works because the member models yield a degree of diversity. Your question is probably unanswerable without the specifics of both those models you put into your ensemble and the dataset in question. $\endgroup$
    – horaceT
    Sep 9, 2016 at 17:25

3 Answers 3

Reset to default
10
+50
$\begingroup$

State-of-the-art algorithms may differ from what is used in production in the industry. Also, the latter can invest in fine-tuning more basic (and often more interpretable) approaches to make them work better than what academics would.

Example 1: According to TechCrunch, Nuance will start using "deep learning tech" in its Dragon speech recognition products this september.

Example 2: Chiticariu, Laura, Yunyao Li, and Frederick R. Reiss. "Rule-Based Information Extraction is Dead! Long Live Rule-Based Information Extraction Systems!." In EMNLP, no. October, pp. 827-832. 2013. https://scholar.google.com/scholar?cluster=12856773132046965379&hl=en&as_sdt=0,22 ; http://www.aclweb.org/website/old_anthology/D/D13/D13-1079.pdf

enter image description here

With that being said:

Which of the ensemble learning algorithms is considered to be state-of-the-art nowadays

One of the state-of-the-art systems for image classification gets some nice gain with ensemble (just like most other systems I far as I know): He, Kaiming, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. "Deep residual learning for image recognition." arXiv preprint arXiv:1512.03385 (2015). https://scholar.google.com/scholar?cluster=17704431389020559554&hl=en&as_sdt=0,22 ; https://arxiv.org/pdf/1512.03385v1.pdf

enter image description here

Improve this answer
$\endgroup$
3
$\begingroup$

I guess one could say that deep learning is pretty much state-of-the-art in most of the subdomains of computer vision (classification, detection, super-resolution, edge detection,...) except for very specific task like SLAM where deep learning is not yet on par with existing methods.

Often to get a few extra percent to win competition networks averaging is used but networks are getting so good that it does not matter that much anymore.

In production it is totally different. Big companies are usually relying on old algorithms that have proven to be effective and that the experts in place have knowledge of and years of practice using them.
Plus integrating a new algorithm in the supply chain requires a lot of time. I think some cameras companies still use Viola Jones detector for face detection and I know for a fact that SIFT is being heavily used in a lot of applications in industry.

They are also still a bit of scepticism towards deep learning methods that are considered dangerous black boxes.
But the impressive results of those algorithms are slowy making people change their minds about it.

Start-ups are more willing to use such solutions as they have to have innovative solutions to get funded.

I would say that in twenty years most of the computer vision based products will use deep learning even if something more effective is discovered in between.
To add to Franck's answer deep learning is changing so fast that ResNets of Kaiming He are not State of the art anymore Densely connected Convolutional Networks and Wide and Deep networks with SGD restarting are now SOTA on EDIT CIFAR and SVHN and probably Imagenet too and even this could change in a few days with ILSVRC 2016 results on the 16th of September.

If you are interested in more state of the art results on MS-COCO the most challenging detection dataset existing will be released at ECCV in October.

Improve this answer
$\endgroup$
4
  • 1
    $\begingroup$ In fact, after double checking, the articles I cited do not mention their results on Imagenet ! So it is my mistake ! but as they are far superior on CIFAR and SVHN I think it must be the same on Imagenet but you never know. I guess they did not mention it to wait for ILSVRC's results but I might be wrong ! $\endgroup$
    – jeandut
    Sep 9, 2016 at 22:36
  • 1
    $\begingroup$ @FranckDernoncourt this frenzy of results is very exciting but also can put a lot of pressure on people who want to publish in this field, which can lead to mistakes like this now infamous SARM article that the author withdrew from NIPS today. $\endgroup$
    – jeandut
    Sep 9, 2016 at 22:41
  • $\begingroup$ Thanks, yes I saw that indeed, but didn't get a chance to check out that paper… I'm having issue emptying my to-read list with all these new ANN PDFs :/ $\endgroup$
    – Franck Dernoncourt
    Sep 9, 2016 at 23:34
  • $\begingroup$ This SARM withdrawal incident makes me rethink the reproducibility crisis in statistics. How much implementation details should be required in the review process, how much is too little, etc. $\endgroup$
    – horaceT
    Sep 10, 2016 at 2:46
2
$\begingroup$

There are a lot of what-ifs involved with your question, and usually finding the best model involves testing most of these on the data. Just because a model in theory could produce more accurate results does not mean it will always produce a model with the lowest error.

That being said... Neural Net ensembles can be very accurate, as long as you can accept the black box. Varying by both number of nodes and number of layers can cover a lot of variance in the data, with introducing this many modelling factors it could be easy to overfit the data.

Random Forests have rarely produced the most accurate results, but boosted trees can model complex relationships like in the AI tasks you discussed without much risk of overfitting.

One would think, well why not just ensemble all of these models together, but this model compromises on the possible strengths of the individual models. Again this would likely lead to some overfitting issues.

Models that are computationally efficient is a different matter, and I would not start with a very complicated neural net. Using a neural net as a benchmark, in my experience it has been most efficient using boosted trees.

This is based on my experience, and a reasonable understanding of the theory underlying each of the modelling types discussed.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.