Predictive models: statistics can't possibly beat machine learning? I am currently following a master program focused on statistics/econometrics. In my master, all students had to do 3 months of research. Last week, all groups had to present their research to the rest of the master students.
Almost every group did some statistical modelling and some machine learning modelling for their research topics and every single time out-of-sample predictions came to talk the simple machine learning models beat the very sophisticated statistical models that every worked on very hard for the last 3 months. No matter how good everyones statistical models get, a simple random forest got lower out-of-sample errors pretty much always.
I was wondering if this is a generally accepted observation? That if it comes to out-of-sample forecasting there is simply no way to beat a simple random forest or extreme gradient boosting model? These two methods are super simple to implement by using R packages, whereas all the statistical models that everyone came up with require quite a lot of skill, knowledge and effort to estimate. 
What are your thoughts of this? Is the only benefit of statistical/econometric models that you gain interpretation? Or were our models just not good enough that they failed to significantly outperform simple random forest predictions? Are there any papers that address this issue? 
 A: It's wrong to state the question the way you worded it. For instance, a significant chunk of machine learning can be called statistical learning. So, your comparison is like apples vs. fruit tarts.
However, I'll go with the way you framed it, and claim the following: when it comes to prediction nothing can be done without some form of statistics because prediction inherently has randomness (uncertainty) in it. Consider this: despite huge success of machine learning in some applications it has absolutely nothing to show off in asset price prediction. Nothing at all. Why? Because in most developed liquid markets asset prices are inherently stochastic. 
You can run machine learning all day long to observe and learn about radioactive decay of atoms, and it will never be able to predict the next atom's decay time, simply because it is random.
As an aspiring statistician it would be foolish on your side to not master machine learning, because it's one of the hottest applications of statistics, unless, of course, you know for sure that you are going to academia. Anyone who's likely to go work in the industry needs to master ML. There is no animosity or competition between statistics and ML crowds at all. In fact, if you like programming you'll feel at home in ML field
A: Statistical modeling is different from machine learning.  For example, a linear regression is both a statistical model and a machine learning model.  So if you compare a linear regression to a random forest, you’re just comparing a simpler machine learning model to a more complicated one. You’re not comparing a statistical model to a machine learning model.
Statistical modeling provides more than interpretation; it actually gives a model of some population parameter.  It depends on a large framework of mathematics and theory, which allows for formulas for things like the variance of coefficients, variance of predictions, and hypothesis testing.  The potential yield of statistical modeling is much greater than machine learning, because you can make strong statements about population parameters instead of just measuring error on holdout, but it’s considerably more difficult to approach a problem with a statistical model.
A: Generally not, but potentially yes under misspecification.  The issue you are looking for is called admissibility.  A decision is admissible if there is no less risky way to calculate it.  
All Bayesian solutions are admissible and non-Bayesian solutions are admissible to the extent that they either match a Bayesian solution in every sample or at the limit.  An admissible Frequentist or Bayesian solution will always beat an ML solution unless it is also admissible.  With that said, there are some practical remarks that make this statement true but vacuous.  
First, the prior for the Bayesian option has to be your real prior and not some prior distribution used to make an editor at a journal happy.  Second, many Frequentist solutions are inadmissible and a shrinkage estimator should have been used instead of the standard solution.  A lot of people are unaware of Stein's lemma and its implications for out of sample error.  Finally, ML can be a bit more robust, in many cases, to misspecification error.
When you move into decision trees and their cousins the forests, you are not using a similar methodology unless you are also using something similar to a Bayes net.  A graph solution contains a substantial amount of implicit information in it, particularly a directed graph.  Whenever you add information to a probabilistic or statistical process you reduce the variability of the outcome and change what would be considered admissible.
If you look at machine learning from a composition of functions perspective, it just becomes a statistical solution but using approximations to make the solution tractable.  For Bayesian solutions, MCMC saves unbelievable amounts of time as does gradient descent for many ML problems.  If you either had to construct an exact posterior to integrate or use brute force on many ML problems, the solar system would have died its heat death before you got an answer.
My guess is that you have a misspecified model for those using statistics, or inappropriate statistics.  I taught a lecture where I proved newborns will float out windows if not appropriately swaddled and where a Bayesian method so radically outperformed a Frequentist method on a multinomial choice that the Frequentist method broke even, in expectation, while the Bayesian method doubled the participants' money.  Now I abused statistics in the former and took advantage of the inadmissibility of the Frequentist estimator in the latter, but a naive user of statistics could easily do what I did.  I just made them extreme to make the examples obvious, but I used absolutely real data.
Random forests are consistent estimators and they seem to resemble certain Bayesian processes.  Because of the linkage to kernel estimators, they may be quite close.  If you see a material difference in performance between solution types, then there is something in the underlying problem that you are misunderstanding and if the problem holds any importance, then you really need to look for the source of the difference as it may also be the case that all models are misspecified.
A: A lot of machine learning might not be that different from p-hacking, for at least some purposes. 
If you test every possible model to find that one that has highest prediction accuracy (historical prediction or out-group prediction) on the basis of historical data, this does not necessarily mean that the results will help to understand what's going on. However, possibly it will find possible relationships that may inform a hypothesis.
Motivating specific hypotheses and then testing them using statistical methods can certainly be similarly p-hacked (or similar) as well.
But the point is that if the criteria is "highest prediction accuracy based on historical data", then there is a high risk of being overconfident in some model that one does not understand, without actually having any idea of what drove those historical results and/or whether they may be informative for the future.
