What is the difference between Econometrics and Machine Learning?

Question

In my understanding, econometrics estimates partial (ceteris paribus) correlations with the aim to primarily estimate causal relations. For that, it normally uses the whole dataset that is available. Econometrics can be parametric and non-parametric.

Meanwhile, machine learning is not interested in causality, but in "fit" with the aim of primarily produce predictions. For that, it normally splits the dataset between the training and the prediction sets. Machine learning can also be parametric and non-parametric.

---

This is what I can make of the core of these two disciplines, but I am sure there is plenty more to it. I am primarily interested in their differences. Can anyone provide a good guide on this please?

Answer 1

First things first. Everything that I say is my understanding only. Hence, as usual, I can be wrong.

Henry is partially right. But Econometrics is also a family of methods. There are a variety of different econometric methods that can be applied depending on the research question at hand as well as the data provided (cross section vs. panel data and so on).

Machine learning in my understanding is a collection of methods which enables machines to learn patterns from past observations (oftentimes in a black box manner). Regression is a standard tool in econometrics as well as machine learning as it allows to learn relationships between variables and to extrapolate these relationships into the future.

Not all econometricians are interested in a causal interpretation of parameters estimates (they rarely can claim a causal interpretation if observational data (non experimental) is used). Oftentimes, like in the case of time series data, econometricians also do only care about predictive performance.

Essentially both are the very same thing but developed in different sub-fields (machine learning being rooted in computer science). They are both a collection of methods. Econometricians also increasingly use machine learning methods like decision trees and neural networks.

You already touched a very interesting point: Causality. Essentially, both fields would like to know the true underlying relationships but as you already mentioned, oftentimes the predictive performance is the main KPI used in machine learning tasks. That is, having a low generalization error is the main goal. Of course, if you know the true causal relationships, this should have the lowest generalization error out of all possible formulations. Reality is very complex and there is no free hunch. Hence, most of the time we have only partial knowledge of the underlying system and sometimes can't even measure the most important influences. But we can use proxy variables that correlate with the true underlying variables we would like to measure.

Long story short and very very superficial: Both fields are related whereas econometricians are mostly interested in finding the true causal relationships (that is, testing some hypothesis) whereas machine learning is rooted rather in the computer science and is mostly interested in building systems with low generalization error.

PS: Using only the whole data set in econometrics should be generally avoided too. Econometricians are getting more aware that relationships found insample do not necessarily generalize to new data. Hence, replication of econometric studies is and always was very important.

Hope this helps in any way.

Answer 2

Some notes in addition to @JustMe:

First, there is a lot of arrogance on both sides of Econometrics and Machine Learning. Discussing which of the two may be a sub-discipline of which is futile. In fact they are both strongly overlapping sub-disciplines of the field of statistics (which is best described as applied mathematics). Both have their own foci and preferences, e.g. Econometrics focus on estimation and testing hypotheses, often in smaller samples, while ML focuses on best functional approximation, often in huge samples. The first focuses on parametric methods making distributional assumptions, the second more often (but by far not exclusively) on non-parametric distribution-free methods. And so on.

Second, if the goal is prediction there is no inherent need to understand causality, as long as random samples from the same population are available. However, understanding causality is of central interest if we want to understand a system (i.e. theory development/testing) or change it (i.e. acting on theory by an intervention). This type of research goal is much more common in econometrics (and other fields like biostatistics) than it is in machine learning.

However, there are machine learning researchers interested in causality as well. The primary difference between fields here are, once again, that econometricians have hypotheses about interventions and try to estimate their effects (e.g. from observational data or experimental data using techniques from causal inference theory such as weighting, matching or selection models) whereas machine learning would rather try to learn causal relationships from the data (e.g. using search algorithms in directed acyclical causal graphs) and the focus is less strongly put on a single intervention.

Answer 3

Something that I think could be stressed more is that econometric modelling often assumes that the model chosen is in fact the true model, in the sense that this model is equivalent to the data generating process (DGP). This is needed to derive powerful distributional results in order to do inference and express uncertainty and make statements such as OLS being the best linear unbiased estimator (BLUE) under standard assumptions. The obtained results are incredibly useful for testing model hypotheses which also helps explain how this framework is so useful for testing economic theory.

On the other hand, machine learning often makes less restrictive assumptions which does not allow for these kind of results and machine learning also focusses more on the approximation error, which is defined as the error between the best predictor in a chosen model and the best predictor among all predictors (often called Bayes predictor). More general "learning guarantees" can be proven which allows to bound the estimation error of the model which very little assumptions. It is then often more natural in machine learning to take a very flexible approach to modelling which is sensible as machine learning researchers often focus on predictive modelling.

I would like to add that advanced methodology in econometrics allows for (among many other things) expressing the model uncertainty e.g. using Bayesian modelling a comparison can be made between the posterior probabilities between candidate models. My point here is that depending on the methodology chosen, econometrics can incorporate more uncertainty then the uncertainty inherent to the parameters and the errors which is often the only uncertainty expressed in many types of analysis.