The Two Cultures: statistics vs. machine learning?

Question

Last year, I read a blog post from Bendan O'Connor entitled "Statistics vs. Machine Learning, fight!" that discussed some of the differences between the two fields. Andrew Gelman responded to favorably to this:

Simon Blomberg:

From R's fortunes package: To paraphrase provocatively, 'machine learning is statistics minus any checking of models and assumptions'. -- Brian D. Ripley (about the difference between machine learning and statistics) useR! 2004, Vienna (May 2004) :-) Season's Greetings!

Andrew Gelman:

In that case, maybe we should get rid of checking of models and assumptions more often. Then maybe we'd be able to solve some of the problems that the machine learning people can solve but we can't!

There was also the "Statistical Modeling: The Two Cultures" paper by Leo Breiman in 2001 which argued that Statisticians rely too heavily on data modeling, and that machine learning techniques are making progress by instead relying on the predictive accuracy of models.

Has the Statistics field changed over the last decade in response to these critiques? Do the two cultures still exist or has Statistics grown to embrace machine learning techniques such as neural networks and support vector machines?

Answer 1

What enforces more separation than there should be is each discipline's lexicon.

There are many instances where ML uses one term and Statistics uses a different term--but both refer to the same thing--fine, you would expect that, and it doesn't cause any permanent confusion (e.g., features/attributes versus expectation variables, or neural network/MLP versus projection-pursuit).

What's much more troublesome is that both disciplines use the same term to refer to completely different concepts.

A few examples:

Kernel Function

In ML, kernel functions are used in classifiers (e.g., SVM) and of course in kernel machines. The term refers to a simple function (cosine, sigmoidal, rbf, polynomial) to map non-linearly separable to a new input space, so that the data is now linearly separable in this new input space. (versus using a non-linear model to begin with).

In statistics, a kernel function is weighting function used in density estimation to smooth the density curve.

Regression

In ML, predictive algorithms, or implementations of those algorithms that return class labels "classifiers" are referred to as machines--e.g., support vector machine, kernel machine. The counterpart to machines are regressors, which return a score (continuous variable)--e.g., support vector regression.

Rarely do the algorithms have different names based on mode--e.g., a MLP is the term used whether it returns a class label or a continuous variable.

In Statistics, regression, if you are attempting to build a model based on empirical data, to predict some response variable based on one or more explanatory variables or more variables--then you are doing regression analysis. It doesn't matter whether the output is a continuous variable or a class label (e.g., logistic regression). So for instance, least-squares regression refers to a model that returns a continuous value; logistic regression on the other hand, returns a class labels.

Bias

In ML, the bias term in the algorithm is conceptually identical to the intercept term used by statisticians in regression modeling.

In Statistics, bias is non-random error--i.e., some phenomenon influenced the entire data set in the same direction, which in turn means that this kind of error cannot be removed by resampling or increasing the sample size.

Answer 2

You run a fancy computer algorithm once -- and you get a CS conference presentation/statistics paper (wow, what a fast convergence!). You commercialize it and run it 1 million times -- and you go broke (ouch, why am I getting useless and irreproducible results all the time???) unless you know how to employ probability and statistics to generalize the properties of the algorithm.

Answer 3

I have spoken on this at a different forum the ASA Statistical Consulting eGroup. My response was more specifically to data mining but the two go hand in hand. We statisticians have snubbed our noses at data miners, computer scientists, and engineers. It is wrong. I think part of the reason it happens is because we see some people in those fields ignoring the stochastic nature of their problem. Some statisticians call data mining data snooping or data fishing. Some people do abuse and misuse the methods but statisticians have fallen behind in data mining and machine learning because we paint them with a broad brush. Some of the big statistical results have come from outside the field of statistics. Boosting is one important example. But statisticians like Brieman, Friedman, Hastie, Tibshirani, Efron, Gelman and others got it and their leadership has brought statisticians into the analysis of microarrays and other large scale inference problems. So while the cultures may never mesh there is now more cooperation and collabortion between the computer scientists, engineers and statisticians.

Answer 4

I think the answer to your first question is simply in the affirmative. Take any issue of Statistical Science, JAMA, Annals of Statistics of the past 10 years and you'll find papers on boosting, SVM, and neural networks, although this area is less active now. Statisticians have appropriated the work of Valiant and Vapnick, but on the other side, computer scientists have absorbed the work of Donoho and Talagrand. I don't think there is much difference in scope and methods anymore. I have never bought Breiman's argument that CS people were only interested in minimizing loss using whatever works. That view was heavily influenced by his participation in Neural Networks conferences and his consulting work; but PAC, SVMs, Boosting have all solid foundations. And today, unlike in 2001, Statistics is more concerned with finite-sample properties, algorithms and massive datasets.

But I think that there are still three important differences that are not going away soon.

1. Methodological Statistics papers are still overwhelmingly formal and deductive, whereas Machine Learning researchers are more tolerant of new approaches even if they don't come with a proof attached;
2. The ML community primarily shares new results and publications in conferences and related proceedings, whereas statisticians use journal papers. This slows down progress in Statistics and identification of star researchers. John Langford a nice post on the subject from a while back;
3. Statistics still covers areas that are (for now) of little concern to ML, such as survey design, sampling, industrial Statistics etc.

Answer 5

Clearly, the two fields clearly face similar but different problems, in similar but not identical ways with analogous but not identical concepts, and work in different departments, journals and conferences.

When I read Cressie and Read's Power Divergence Statistic it all snapped into place for me. Their formula generalizes commonly used test statistics into one that varies by one exponent, lamda. There are two special cases, lamda=0 and lamda=1.

Computer Science and Statistics fit along a continuum (that presumably could include other points). At one value of lamda, you get statistics commonly cited in Statistics circles, and at the other you get statistics commonly cited in Comp Sci circles.

Statistics

• Lamda = 1
• Sums of squares appear a lot
• Variance as a measure of variability
• Covariance as a measure of association
• Chi-squared statistic as a measure of model fit

Computer science:

• Lamda = 0
• Sums of logs appear a lot
• Entropy as a measure of variability
• Mutual information as a measure of association
• G-squared statistic as a measure of model fit

Answer 6

The finer composer, it is felt, should be capable of elaborating the work [of music] solely in his head, and ought not to need the crutch of trying it out at the keyboard.
— Charles Rosen, Piano Notes

Answer 7

The biggest difference I see between the communities is that statistics emphasizes inference, whereas machine learning emphasized prediction. When you do statistics, you want to infer the process by which data you have was generated. When you do machine learning, you want to know how you can predict what future data will look like w.r.t. some variable.

Of course the two overlap. Knowing how the data was generated will give you some hints about what a good predictor would be, for example. However, one example of the difference is that machine learning has dealt with the p >> n problem (more features/variables than training samples) since its infancy, whereas statistics is just starting to get serious about this problem. Why? Because you can still make good predictions when p >> n, but you can't make very good inferences about what variables are actually important and why.

Answer 8

Ideally one should have a thorough knowledge of both statsitics and machine learning before attempting to answer his question. I am very much a neophyte to ML, so forgive me if wat I say is naive.

I have limited experience in SVMs and regression trees. What strikes me as lacking in ML from a stats point of view is a well developed concept of inference.

Inference in ML seems to boil down almost exclusively to the predictice accuracy, as measured by (for example) mean classification error (MCE), or balanced error rate (BER) or similar. ML is in the very good habit of dividing data randomly (usually 2:1) into a training set and a test set. Models are fit using the training set and performance (MCE, BER etc) is assessed using the test set. This is an excellent practice and is only slowly making its way into mainstream statistics.

ML also makes heavy use of resampling methods (especially cross-validation), whose origins appear to be in statistics.

However, ML seems to lack a fully developed concept of inference - beyond predictive accuracy. This has two results.

1) There does not seem to be an appreciation that any prediction (parameter estimation etc.) is subject to a random error and perhaps systemmatics error (bias). Statisticians will accept that this is an inevitable part of prediction and will try and estimate the error. Statistical techniques will try and find an estimate that has minimum bias and random error. Their techniques are usually driven by a model of the data process, but not always (eg. Bootstrap).

2) There does not seem to be a deep understanding in ML of the limits of applying a model to new data to a new sample from the same population (in spite of what I said earlier about the training-test data set approach). Various statistical techniques, among them cross validation and penalty terms applied to likelihood-based methods, guide statisticians in the trade-off between parsimony and model complexity. Such guidelines in ML seem much more ad hoc.

I've seen several papers in ML where cross validation is used to optimise a fitting of many models on a training dataset - producing better and better fit as the model complexity increases. There appears little appreciation that the tiny gains in accuracy are not worth the extra complexity and this naturally leads to over-fitting. Then all these optimised models are applied to the test set as a check on predictive performance and to prevent overfitting. Two things have been forgotten (above). The predictive performance will have a stochastic component. Secondly multiple tests against a test set will again result in over-fitting. The "best" model will be choisen by the ML practitioner without a full appreciation he/she has cherry picked from one realisation of many possible outomes of this experiment. The best of several tested models will almost certainly not reflect the true performance on new data.

Any my 2 cents worth. We have much to learn from each other.

Answer 9

I don't really know what the conceptual/historical difference between machine learning and statistic is but I am sure it is not that obvious... and I am not really interest in knowing if I am a machine learner or a statistician, I think 10 years after Breiman's paper, lots of people are both...

Anyway, I found interesting the question about predictive accuracy of models. We have to remember that it is not always possible to measure the accuracy of a model and more precisely we are most often implicitly making some modeling when measuring errors.

For Example, mean absolute error in time series forecast is a mean over time and it measures the performance of a procedure to forecast the median with the assumption that performance is, in some sense, stationary and shows some ergodic property. If (for some reason) you need to forecast the mean temperature on earth for the next 50 years and if your modeling performs well for the last 50 years... it does not means that...

More generally, (if I remember, it is called no free lunch) you can't do anything without modeling... In addition, I think statistic is trying to find an answer to the question : "is something significant or not ", this is a very important question in science and can't be answered through a learning process. To state John Tukey (was he a statistician ?) :

The combination of some data and an aching desire for an answer does not ensure that a reasonable answer can be extracted from a given body of data

Hope this helps !

Answer 10

I disagree with this question as it suggests that machine learning and statistics are different or conflicting sciences.... when the opposite is true!

machine learning makes extensive use of statistics... a quick survey of any Machine learning or data mining software package will reveal Clustering techniques such as k-means also found in statistics.... will also show dimension reduction techniques such as Principal components analysis also a statistical technique... even logistic regression yet another.

In my view the main difference is that traditionally statistics was used to proove a pre conceived theory and usually the analysis was design around that principal theory. Where with data mining or machine learning the opposite approach is usually the norm in that we have the outcome we just want to find a way to predict it rather than ask the question or form the theory is this the outcome!

Answer 11

In such a discussion, I always recall the famous Ken Thompson quote

When in doubt, use brute force.

In this case, machine learning is a salvation when the assumptions are hard to catch; or at least it is much better than guessing them wrong.

Answer 12

Machine Learning seems to have its basis in the pragmatic - a Practical observation or simulation of reality. Even within statistics, mindless "checking of models and assumptions" can lead to discarding methods that are useful.

For example, years ago, the very first commercially available (and working) Bankruptcy model implemented by the credit bureaus was created through a plain old linear regression model targeting a 0-1 outcome. Technically, that's a bad approach, but practically, it worked.