15
$\begingroup$

I am currently following a master program focused on statistics/econometrics. In my master, all students had to do 3 months of research. Last week, all groups had to present their research to the rest of the master students.

Almost every group did some statistical modelling and some machine learning modelling for their research topics and every single time out-of-sample predictions came to talk the simple machine learning models beat the very sophisticated statistical models that every worked on very hard for the last 3 months. No matter how good everyones statistical models get, a simple random forest got lower out-of-sample errors pretty much always.

I was wondering if this is a generally accepted observation? That if it comes to out-of-sample forecasting there is simply no way to beat a simple random forest or extreme gradient boosting model? These two methods are super simple to implement by using R packages, whereas all the statistical models that everyone came up with require quite a lot of skill, knowledge and effort to estimate.

What are your thoughts of this? Is the only benefit of statistical/econometric models that you gain interpretation? Or were our models just not good enough that they failed to significantly outperform simple random forest predictions? Are there any papers that address this issue?

Improve this question
$\endgroup$
10
  • 5
    $\begingroup$ This may well be closed as "too broad". (Hopefully not as "opinion-based"!) My take: I don't think there is a universal answer. My experience is that statistical models are better if there are fewer observations, because then imposing some kind of structure improves on a largely model-free approach. Conversely, RFs are better if there are many observations. ... $\endgroup$
    – Stephan Kolassa
    Mar 24, 2018 at 12:11
  • 4
    $\begingroup$ ... The other question is what exactly was evaluated, and how. If point predictions were evaluated appropriately (accuracy measures can be surprisingly misleading), that is a different matter than if density predictions were. Statistical models may be better at density forecasts, again because you need lots more data. $\endgroup$
    – Stephan Kolassa
    Mar 24, 2018 at 12:12
  • 1
    $\begingroup$ @StephanKolassa: I think a good answer (or set of several answers) to this question would comprise reasons why there isn't a universal answer - theoretically & practically - , how predictive performance is evaluated, how to draw a distinction between statistical & machine learning methods, what goals there might be beyond prediction, & a couple of things I haven't thought of. So a wide scope; but not too broad in my opinion, & trying to limit it might just preclude the making of useful general points. $\endgroup$
    – Scortchi - Reinstate Monica ♦
    Mar 24, 2018 at 13:48
  • 5
    $\begingroup$ What we don't want is a collection of anecdotes - I urge users to flag for deletion answers that come to little more than e.g. "I've always found that random forests beat logistic regression", however wordy. We can be a bit slacker about comments, but long threads will be moved to chat. $\endgroup$
    – Scortchi - Reinstate Monica ♦
    Mar 24, 2018 at 13:51
  • 14
    $\begingroup$ I don't think that there's a meaningful distinction between statistics and machine learning. For example, Leo Breiman, a prominent random forest researcher, was a professor of statistics at UC Berkeley. In the context of your anecdote, RF happened to be better than the other models people had fit, but I see no reason that this must be true in general (see also the No Free Lunch theorem). Perhaps this says more about the data set (or even the students) than the methods. $\endgroup$
    – Sycorax ♦
    Mar 24, 2018 at 15:23

4 Answers 4

Reset to default
21
$\begingroup$

Statistical modeling is different from machine learning. For example, a linear regression is both a statistical model and a machine learning model. So if you compare a linear regression to a random forest, you’re just comparing a simpler machine learning model to a more complicated one. You’re not comparing a statistical model to a machine learning model.

Statistical modeling provides more than interpretation; it actually gives a model of some population parameter. It depends on a large framework of mathematics and theory, which allows for formulas for things like the variance of coefficients, variance of predictions, and hypothesis testing. The potential yield of statistical modeling is much greater than machine learning, because you can make strong statements about population parameters instead of just measuring error on holdout, but it’s considerably more difficult to approach a problem with a statistical model.

Improve this answer
$\endgroup$
6
  • 1
    $\begingroup$ As far as I understand you say that with statistics you get more benefits like the variance of coefficients, variance of predictions, and hypothesis testing. But when it purely comes to predictive modeling, i.e. making point forecasts of some response variable, do you think the statistical models can beat the machine learning models? $\endgroup$
    – dubvice
    Mar 24, 2018 at 14:54
  • 6
    $\begingroup$ This is the answer (+1!). In my view (and perhaps of others as well) there are several types of statistical analyses: descriptive, inferential, predictive, exploratory, etc. Machine learning would mostly fall within predictive analysis, and most of it doesn't allow you to make inferential assertions on things, so it all boils down to "use the right tool for the job at hand" (given the linear regression example, it can be used in all fields, e.g. estimating conditional expectations, which is a descriptive task). $\endgroup$
    – Firebug
    Mar 24, 2018 at 15:46
  • 2
    $\begingroup$ This sounds like the assertion that standard statistical modeling can be better for inference (as opposed to prediction) than machine learning, which can help model interpretability. While it’s certainly true if we compare a ordinary least squares regression to a deep neural network, given that the original question specifically references random forest (a good ML algorithm for inference), such an assertion is a bit fuzzy. $\endgroup$
    – Greenstick
    Mar 24, 2018 at 15:50
  • 3
    $\begingroup$ Here is some solid evidence from the time series domain where statistical models consistently beat machine learning approaches: Makridakis "Statistical and Machine Learning forecasting methods: Concerns and ways forward". $\endgroup$
    – Richard Hardy
    Mar 26, 2018 at 9:10
  • 1
    $\begingroup$ That is just the perfect answer. Here is an example: say you have a measure that predicts survival of patients with a given disease. There are international standards on how to define if this measure in clinically valid (basically if the coefficient is different from 0 with a pvalue below 5% in a univariate or multivariate model). Although I am absolutely sure that 99% of the time a random forest with sufficient data would be a way better predicting model. $\endgroup$
    – Rémy Nicolle
    Mar 28, 2018 at 6:32
5
$\begingroup$

It's wrong to state the question the way you worded it. For instance, a significant chunk of machine learning can be called statistical learning. So, your comparison is like apples vs. fruit tarts.

However, I'll go with the way you framed it, and claim the following: when it comes to prediction nothing can be done without some form of statistics because prediction inherently has randomness (uncertainty) in it. Consider this: despite huge success of machine learning in some applications it has absolutely nothing to show off in asset price prediction. Nothing at all. Why? Because in most developed liquid markets asset prices are inherently stochastic.

You can run machine learning all day long to observe and learn about radioactive decay of atoms, and it will never be able to predict the next atom's decay time, simply because it is random.

As an aspiring statistician it would be foolish on your side to not master machine learning, because it's one of the hottest applications of statistics, unless, of course, you know for sure that you are going to academia. Anyone who's likely to go work in the industry needs to master ML. There is no animosity or competition between statistics and ML crowds at all. In fact, if you like programming you'll feel at home in ML field

Improve this answer
$\endgroup$
2
$\begingroup$

Generally not, but potentially yes under misspecification. The issue you are looking for is called admissibility. A decision is admissible if there is no less risky way to calculate it.

All Bayesian solutions are admissible and non-Bayesian solutions are admissible to the extent that they either match a Bayesian solution in every sample or at the limit. An admissible Frequentist or Bayesian solution will always beat an ML solution unless it is also admissible. With that said, there are some practical remarks that make this statement true but vacuous.

First, the prior for the Bayesian option has to be your real prior and not some prior distribution used to make an editor at a journal happy. Second, many Frequentist solutions are inadmissible and a shrinkage estimator should have been used instead of the standard solution. A lot of people are unaware of Stein's lemma and its implications for out of sample error. Finally, ML can be a bit more robust, in many cases, to misspecification error.

When you move into decision trees and their cousins the forests, you are not using a similar methodology unless you are also using something similar to a Bayes net. A graph solution contains a substantial amount of implicit information in it, particularly a directed graph. Whenever you add information to a probabilistic or statistical process you reduce the variability of the outcome and change what would be considered admissible.

If you look at machine learning from a composition of functions perspective, it just becomes a statistical solution but using approximations to make the solution tractable. For Bayesian solutions, MCMC saves unbelievable amounts of time as does gradient descent for many ML problems. If you either had to construct an exact posterior to integrate or use brute force on many ML problems, the solar system would have died its heat death before you got an answer.

My guess is that you have a misspecified model for those using statistics, or inappropriate statistics. I taught a lecture where I proved newborns will float out windows if not appropriately swaddled and where a Bayesian method so radically outperformed a Frequentist method on a multinomial choice that the Frequentist method broke even, in expectation, while the Bayesian method doubled the participants' money. Now I abused statistics in the former and took advantage of the inadmissibility of the Frequentist estimator in the latter, but a naive user of statistics could easily do what I did. I just made them extreme to make the examples obvious, but I used absolutely real data.

Random forests are consistent estimators and they seem to resemble certain Bayesian processes. Because of the linkage to kernel estimators, they may be quite close. If you see a material difference in performance between solution types, then there is something in the underlying problem that you are misunderstanding and if the problem holds any importance, then you really need to look for the source of the difference as it may also be the case that all models are misspecified.

Improve this answer
$\endgroup$
1
$\begingroup$

A lot of machine learning might not be that different from p-hacking, for at least some purposes.

If you test every possible model to find that one that has highest prediction accuracy (historical prediction or out-group prediction) on the basis of historical data, this does not necessarily mean that the results will help to understand what's going on. However, possibly it will find possible relationships that may inform a hypothesis.

Motivating specific hypotheses and then testing them using statistical methods can certainly be similarly p-hacked (or similar) as well.

But the point is that if the criteria is "highest prediction accuracy based on historical data", then there is a high risk of being overconfident in some model that one does not understand, without actually having any idea of what drove those historical results and/or whether they may be informative for the future.

Improve this answer
$\endgroup$

Not the answer you're looking for? Browse other questions tagged or ask your own question.