11
$\begingroup$

In my understanding, econometrics estimates partial (ceteris paribus) correlations with the aim to primarily estimate causal relations. For that, it normally uses the whole dataset that is available. Econometrics can be parametric and non-parametric.

Meanwhile, machine learning is not interested in causality, but in "fit" with the aim of primarily produce predictions. For that, it normally splits the dataset between the training and the prediction sets. Machine learning can also be parametric and non-parametric.


This is what I can make of the core of these two disciplines, but I am sure there is plenty more to it. I am primarily interested in their differences. Can anyone provide a good guide on this please?

$\endgroup$
9
  • 12
    $\begingroup$ I would guess that econometrics is a subject or discipline while machine learning is a family of methods. You can use machine learning in econometrics, as well as in other subject areas, and you can use other methods in econometrics $\endgroup$ – Henry Aug 7 '17 at 8:05
  • 6
    $\begingroup$ Ragnar Frisch wrote in the first issue of Econometrica that econometrics is about: "economic theory in its relation to statistics and mathematics". Thus, econometric models often come out of economic theory or mathematics/game theory before they are fitted to data. Statistics and machine learning, however, are data-driven and not theory-driven. In my opinion, this is the most crucial difference. And in econometrics, there are also predictive models (in contrast to explanatory models) or flexible models (needing training/validation). But they are just part of the methods used. $\endgroup$ – Achim Zeileis Aug 7 '17 at 8:16
  • 1
    $\begingroup$ what is the difference between a banana and fruits? econometrics is a subfield of machine learning... $\endgroup$ – Antoine Aug 7 '17 at 10:10
  • 3
    $\begingroup$ @Antoine I would rather argue that Machine Learning is a subfield of Econometrics... $\endgroup$ – rbm Aug 7 '17 at 11:31
  • 6
    $\begingroup$ @Antoine you can't be serious. I have never seen the problem of statistical testing in a book that labeled itself "machine learning" $\endgroup$ – tomka Aug 7 '17 at 12:22
4
$\begingroup$

First things first. Everything that I say is my understanding only. Hence, as usual, I can be wrong.

Henry is partially right. But Econometrics is also a family of methods. There are a variety of different of econometric methods that can be applied depending on the research question at hand as well as the data provided (cross section vs. panel data and so on).

Machine learning in my understanding is a collection of methods which enables machines to learn patterns from past observations (oftentimes in a black box manner). Regression is a standard tool in econometrics as well as machine learning as it allows to learn relationships between variables and to extrapolate these relationships into the future.

Not all econometricians are interested in a causal interpretation of parameters estimates (they rarely can claim a causal interpretation if observational data (non experimental) is used). Oftentimes, like in the case of time series data, econometricians also do only care about predictive performance.

Essentially both are the very same thing but developed in different sub-fields (machine learning being rooted in computer science). They are both a collection of methods. Econometricians also increasingly use machine learning methods like decision trees and neural networks.

You already touched a very interesting point: Causality. Essentially, both fields would like to know the true underlying relationships but as you already mentioned, oftentimes the predictive performance is the main KPI used in machine learning tasks. That is, having a low generalization error is the main goal. Of course, if you know the true causal relationships, this should have the lowest generalization error out of all possible formulations. Reality is very complex and there is no free hunch. Hence, most of the time we have only partial knowledge of the underlying system and sometimes can't even measure the most important influences. But we can use proxy variables that correlate with the true underlying variables we would like to measure.

Long story short and very very superficial: Both fields are related whereas econometricians are mostly interested in finding the true causal relationships (that is, testing some hypothesis) whereas machine learning is rooted rather in the computer science and is mostly interested in building systems with low generalization error.

PS: Using only the whole data set in econometrics should be generally avoided too. Econometricians are getting more aware that relationships found insample do not necessarily generalize to new data. Hence, replication of econometric studies is and always was very important.

Hope this helps in any way.

$\endgroup$
4
  • $\begingroup$ Achim's commented that ML is little (if not at all) driven by theory, which is not what you say. Can you expand on this? That is probably a core element of the analysis. $\endgroup$ – luchonacho Aug 7 '17 at 11:00
  • $\begingroup$ @luchonacho this is addressed in my note on this answer $\endgroup$ – tomka Aug 7 '17 at 11:53
  • $\begingroup$ @luchonacho tomka made a very good point there. Still, I want to stress the fact (and tomka also mentioned this already) that both fields should and are interested in causality. In maschine learning terms: Knowing the true causal relationships should result in a model having the lowest bias as well as variance. A really important step in any maschine learning apllication is the feature feature engineering step and this should be driven by theory regardless of what field you are from. $\endgroup$ – JustMe Aug 7 '17 at 12:46
  • $\begingroup$ small thing to add (which you touched on in the paragraph before the ps): differences in the concern for where bias located. In ML, concern for bias in the model (eg rmse); in econometrics, concern for bias in the parameter of interest and potential for omitted variables bias. Generally speaking, of course. $\endgroup$ – paqmo Jan 28 '18 at 13:58
2
$\begingroup$

Some notes in addition to @JustMe:

First, there is a lot of arrogance on both sides of Econometrics and Machine Learning. Discussing which of the two may be a sub-discipline of which is futile. In fact they are both strongly overlapping sub-disciplines of the field of statistics (which is best described as applied mathematics). Both have their own foci and preferences, e.g. Econometrics focus on estimation and testing hypotheses, often in smaller samples, while ML focuses on best functional approximation, often in huge samples. The first focuses on parametric methods making distributional assumptions, the second more often (but by far not exclusively) on non-parametric distribution-free methods. And so on.

Second, if the goal is prediction there is no inherent need to understand causality, as long as random samples from the same population are available. However, understanding causality is of central interest if we want to understand a system (i.e. theory development/testing) or change it (i.e. acting on theory by an intervention). This type of research goal is much more common in econometrics (and other fields like biostatistics) than it is in machine learning.

However, there are machine learning researchers interested in causality as well. The primary difference between fields here are, once again, that econometricians have hypotheses about interventions and try to estimate their effects (e.g. from observational data or experimental data using techniques from causal inference theory such as weighting, matching or selection models) whereas machine learning would rather try to learn causal relationships from the data (e.g. using search algorithms in directed acyclical causal graphs) and the focus is less strongly put on a single intervention.

$\endgroup$
0
$\begingroup$

Something that I think could be stressed more is that econometric modelling often assumes that the model chosen is in fact the true model, in the sense that this model is equivalent to the data generating process (DGP). This is needed to derive powerful distributional results in order to do inference and express uncertainty and make statements such as OLS being the best linear unbiased estimator (BLUE) under standard assumptions. The obtained results are incredibly useful for testing model hypotheses which also helps explain how this framework is so useful for testing economic theory.

On the other hand, machine learning often makes less restrictive assumptions which does not allow for these kind of results and machine learning also focusses more on the approximation error, which is defined as the error between the best predictor in a chosen model and the best predictor among all predictors (often called Bayes predictor). More general "learning guarantees" can be proven which allows to bound the estimation error of the model which very little assumptions. It is then often more natural in machine learning to take a very flexible approach to modelling which is sensible as machine learning researchers often focus on predictive modelling.

I would like to add that advanced methodology in econometrics allows for (among many other things) expressing the model uncertainty e.g. using Bayesian modelling a comparison can be made between the posterior probabilities between candidate models. My point here is that depending on the methodology chosen, econometrics can incorporate more uncertainty then the uncertainty inherent to the parameters and the errors which is often the only uncertainty expressed in many types of analysis.

$\endgroup$
1
  • $\begingroup$ I would disagree with a big chunk of this, often quite strongly. By this kind of logic, you could say that a hammer requires a longer handle than a screwdriver, and so a screwdriver is a "better" tool. I agree that you generally need more assumptions to get E[y | do(x)] than to get E[y | x], but if the goals are fundamentally different, how can you compare the assumption costs? $\endgroup$ – Dimitriy V. Masterov May 5 at 22:14

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.