I am looking at a setting where you only have observational data, i.e., non-experimental data and all the predictors $\{X_k\}_{k=1}^p$ are not manipulable, i.e., you view the data set $\{(y_i,\mathbf{X}_i)\}_{i=1}^n$ as $n$ i.i.d. draw from the distribution $P_{\mathbf{x},y}$.

I was told that machine learning primarily focus on prediction and may not care about inference----how $y$ changes as some $X_i$ changes.

My first question is: in this setting, is knowing inference useful? As none of the predictors are under the researcher's control, non-manipulable, I feel that knowing how $y$ changes as some $X_i$ changes is not that useful. And you cannot draw any causal relation, so I guess in this setting, the goal is probably just prediction. You can argue there are some structural relationship but you cannot test that.

Let's say for some reason we still want to do inference, which is also listed as one of the difference between machine learning and econometric. First, it's hard to tell "how $y$ changes as some $X_i$ changes". But there is a well-defined notion of "how $y$ changes as some $X_i$ changes on average", which is closely related to the conditional mean $E[Y|X]$ (with this, we can derive the partiall effect of $x_i$ on $E[y]$). Note that if we have enough data, then we can recover $E[Y|X]$ using data non-parametrically. If we know $E[Y|X]$, then the inference problem is solved. (A digress, "big data" is a relative notion, if the dimension of $\mathbf{X}$ is large, then we need big Big BIG data to estimate $E[Y|X]$ robustly. That's why we still need models like neural nets rather than just non-parametrically estimate $E[Y|X]$, since we can't always do that).

My second question is: in econometrics, in most cases, we write down a linear model $y_i=\mathbf{\beta'x_i}+u_i$. Then the textbook spent a lot of ink talk about consistency of OLS estimator and endogenity. Here are a set of specific questions:

  1. let's say the data is really generated from $y_i=\mathbf{\beta'x_i}+u_i$ and further, assume $E[\mathbf{x}_iu_i]=0$ (which is not necessarily $E[u_i|\mathbf{x}_i]=0$), then we know OLS estimator of $\beta$ is consistent (no endogenity issue). However, in this case, what is the interpretation of $\beta$? Note that $E[y|\mathbf{x}]=\beta'\mathbf{x}+E[u_i|\mathbf{x}]\neq \beta'\mathbf{x}$. In this case, even if OLS will give you the "true $\beta$" in the limit, but this true $\beta$ does not help you to do inference. This is an interesting case, you can cook the data such that $E[\mathbf{x}_iu_i]=0$ but $E[u_i|\mathbf{x}_i]\neq 0$, in this case, OLS estimator is pretty close to the true $\beta$ (assume large dataset), then the residual $\hat{e}_i\equiv y_i-\hat{\beta}'_{ols}\mathbf{x_i}$ is very close to $u_i$. Hence $E[u_i|\mathbf{x}_i]\neq 0$ means that for some values of $x_i$ (think about $\mathbf{x}$ is one dimension), $u_i$ is negative and for some value of $x_i$, $u_i$ is positive (imagine a U shape pattern fit with a line). Then you probably want to add a quadratic term in your original linear regression model, since that will probably better capture the pattern in the data. However, as I cook the data in this way such that the true model is linear and your OLS estimator is consistent. Everything is perfect, but still a "wrong" model (i.e., adding a quadratic term) will probably have a better predictive power than your true linear model. My question is, in practice, this could also occur, then how can you tell whether a "U" shape data pattern is not coming from a linear model used by nature with $E[\mathbf{x}_iu_i]=0$ but $E[u_i|\mathbf{x}_i]\neq 0$? This is an example where nature (you) generates data using a linear model but with specific joint distribution of $x_i$ and $u_i$ such that $E[\mathbf{x}_iu_i]=0$ but $E[u_i|\mathbf{x}_i]\neq 0$, which makes the data looks like $U$ shape and hence the conditional expectation $E[Y|X]$ is better captured by a quadratic model, i.e., $E[Y|X]=a+bx_1+c x^2_1$. Note that if you use OLS to run the linear regression with 2 predictors, $x_1$ and $x^2_1$, then you get residual $\hat{e}'_i\equiv y_i-\hat{a}_{ols}-\hat{b}_{ols}x_i-\hat{c}_{ols}x^2_1$, you can check whether $E[\hat{e}'_i|x_i]=0$ or not. If it is true, then I would say your model capture the functional relation $E[y|x]$. Then we don't have to worry about endogeneity issue, since the goal is to estimate $E[y|x]$ correctly. In this example, running $y_i=\beta_0+\beta_1x+u_i$ will not have endogeneity issue but it is a terrible estimator of $E[y|x]$. I call this case, you win a battle by running the true linear model with just $x$ and consistently estimated your $\beta$'s but lose a war, since the true $\beta$'s in the case are not helpful to explain $E[y|x]$.

  2. how to check whether $E[u_i\mathbf{x}_i]=0$ or $E[u_i|\mathbf{x}_i]$? I don't think that there is a way to do that since $u_i$ is unobservable. (note that here $u_i$ is NOT the residual after you run the OLS and use $y_i$ to subtract $\hat{\beta}'_{ols}\mathbf{x}_i$. $u_i$ comes from the true model $y_i=\beta'\mathbf{x}_i+u_i$ and you need to figure out $\beta$ from data.) My professor said that "If a reviewer just wants to reject an empirical paper anyhow, then the reviewer will just raise the question: did you check the endogenity". I guess this is as I just explained, i.e., there is no way to check whether $E[u_i\mathbf{x}_i]=0$ or $E[u_i|\mathbf{x}_i]$. You can use your domain knowledge but you never know what's has been left into $u_i$. Then my question is: when you read a paper show your R summary table with all those $\beta$'s, how should you trust those $\beta$'s and how should you interpret those $\beta$'s, like in question 1, if just $E[u_iX_i]= 0$ but if $E[u_i|X_i]\neq 0$, then even if you have "true $\beta$", that $\beta$ is not the partial effect of $\mathbf{x}$ on $y$. And if $E[u_iX_i]\neq 0$ (which you never know), then you are not even obtaining the "true $\beta$" but OLS will always produce something, then how should we interpret that something? Like in question 1, a U shape data pattern fitted with a line, then what's the meaning of the slope?

  3. liner model gives great interpretability which machine learning model usually don't have, but relies on very strong assumptions that is not testable. In practice, how should we make use of those linear model wisely and with cautious?

In practice, what I will do is the following: with the data set, I will try many different types of transformation of the predictors and/or the response variable such as take high order or log or whatever and then call the set of transformed predictors $\mathbf{w}$, then I will run a linear regression of possibly transformed $y$ over $\mathbf{w}$, then I get my residuals $\hat{e}_i=y_i-\beta'\mathbf{w}$, then I check whether $E[\hat{e}_i|\mathbf{w}]=0$ (if it is possible, I check it for each value of vector $\mathbf{w}$), then if it is roughly true, I claim that $E[y|\mathbf{w}]=\beta'\mathbf{w}$ and there will be no endogeneity issues. But it seems that this method has a problem. If nature does generate the data using model $y_i=\beta_0+\beta_1x_i+u_i$such that $x_i$ and $u_i$ are correlated, then I wonder if there is always a transformation of $x_i$ (call it $w_i$) and/or $y_i$ such that after we run OLS for the transformed data, we can get $E[\hat{e}_i|w_i]=0$? I guess not.

  • $\begingroup$ Inference means drawing conclusions on a general population using just a sample of that population. Its major tools are p-values, confidence intervals and estimates. There is no direct relation between inference and studying effects of some variable(s) on another. We can do inference on (true) effects, but we can also study effects in a purely descriptive way. In machine learning, while the main focus is often on predictions, calculation or visualization of effects is also quite prominently done, depending on the objectives of the analysis. $\endgroup$
    – Michael M
    Commented Sep 30, 2016 at 7:26
  • $\begingroup$ @MichaelM While one can do inference that is purely descriptive, I think you can also think of doing inference about a hyper-population that consists of the data that we can observe and sample, at least in principle, and some data that we cannot observe or sample, say the wages of college graduates if they had not gone to college and the wage of HS graduates if they had gone to college. In this setting, there is a direct relation between inference and causality. $\endgroup$
    – dimitriy
    Commented Sep 30, 2016 at 16:36
  • $\begingroup$ @DimitriyV.Masterov I am reading the book "Most harmless econometrics" and I like that book. In that book, it tries to argue that the ultimate goal or ideal goal for running these regression is to claim causality. I feel that if all your predictors are dummies or if your model is saturated, then I feel linear regression makes sense in this setting and the coefficients have real meaning, i.e., the conditional expectation, and if on top of that, your data is experimental where you can claim no endogeneity, then the coefficients in the regression actually mean counterfactual effect. $\endgroup$
    – KevinKim
    Commented Sep 30, 2016 at 17:11
  • $\begingroup$ @DimitriyV.Masterov In the saturated model, the conditional expectation is linear so the only thing left is checking endogeneity. But if the mode is not saturated, then it is hard to interpret the coefficient in the same way. E.g., in the wage example, if you put years of education as a predictor in a linear regression, how much do you believe that linearity assumption is still correct? And it is clearly not experimental data, but you can run OLS anyway, then how should I interpret the coefficient of this variable in the linear regression? And how much should I trust the results? $\endgroup$
    – KevinKim
    Commented Sep 30, 2016 at 17:14
  • $\begingroup$ My example was with dummies, so fully saturated. I would definitely not interpret this regression in a causal sense, merely as descriptive. You would need something more sophisticated to do the former. I was merely disputing the claim that there is no tight connection between inference and causality. $\endgroup$
    – dimitriy
    Commented Sep 30, 2016 at 17:36

1 Answer 1


I try to answer my own question in the following way:

Note that if the nature generates data using model $y_i=\beta_0+\beta_1x_i+u_i$, then either there are endogeneity ($E[ux]\neq 0$) or no endogeneity ($E[ux]= 0$).

  1. if there are endogenity, i.e., $E[ux]\neq 0$, then clearly $E[u|x]\neq 0$ (which you can never test via data since $u$ is not observable), then $E[Y|X]\neq\beta_0+\beta_1x$. So in this case, even if somehow you know the true $\beta_0$ and $\beta_1$, then it does not help you to learn $E[Y|X]$. Nevertheless, in this case with endogeneity, if you run OLS, you still may recover $E[y|x]$, e.g., $y_i=3+5x_i+u_i$, where $x_i\in\{0,1\}$ and nature generates $u_i$ such that when $x_i=0$, $u_i$ are all positive and when $x_i=1$, $u_i$ are all negative. In this case, OLS will not give you true $\beta_0=3,\beta_1=5$, but OLS will give you $E[y|x]=\hat{a}_{ols}+\hat{b}_{ols}x$, where $\hat{a}_{ols}\approx E[Y|X=0]$ and $\hat{b}_{ols}\approx E[Y|X=1]-E[Y|X=0]$ for big data set. And if you run OLS, you get your residual $\hat{e}_i$ then you check $E[\hat{e}_i|x_i]$, then you probably will find out this is 0 (since in this example, $x$ is dummy, so it is for sure that this is 0, if $x$ is not dummy, then not sure, but this is something that you can check.)

  2. if $E[uX]=0$ (no endogeneity, but again, you can't test it by data), then it could be $E[u|X]=0$ or $E[u|X]\neq 0$.

    (2a). if it is $E[u|X]=0$, then the ols you run will lead to those true $\beta$'s and $E[y|x]=\beta_0+\beta_1x$ and you can check your ols residual $E[\hat{e}|x]$, which is probably 0.

    (2b). if it is $E[u|X]\neq 0$, then your ols is still consistent, i.e., you will get true $\beta$'s but those true $\beta$'s cannot represent $E[y|x]$.

In summary, you never know whether $E[ux]\neq 0$ or $E[ux]= 0$ (but $E[u|x]\neq 0$ or $E[u|x]= 0$). Hence, if in practice, I run an OLS, (which means I assume the data is generated by $y_i=\beta_0+\beta_1x_i+u_i$), then

(1) I have no idea and cannot test whether my OLS estimates are close to $\beta$'s in my model;

(2) But I can check whether $\hat{a}_{ols}+\hat{b}_{ols}x$ (note that here $\hat{a}_{ols}$ and $\hat{b}_{ols}$ may be far away from the true $\beta_0,\beta_1$) captures the conditional expectation. Because if (2) is true, then it means we can write $y=a+bx+e$, where $a,b$ are the limits of $\hat{a}_{ols},\hat{b}_{ols}$ and $E[y|x]=a+bx+E[e|x]$ and $E[e|x]=0$, this is something I can check from residual $E[\hat{e}|x]=0$. So if after I check residual and find $E[\hat{e}|x]=0$, then I can claim $E[y|x]=a+bx$ and then I don't care about what is true $\beta_0,\beta_1$, since my goal is to learn the conditional mean. But if I check residual and find $E[\hat{e}|x]\neq 0$, then this just means that the conditional mean (which has a true functional form in principle) is not captured by my OLS expression $a+bx$. In this case, it is more likely that the conditional mean just not in the linear form of the current set of predictors (may be after you transfer some predictors, then it becomes linear), i.e., the linearity assumption of the conditional mean is questionable. It is less likely that the conditional mean is still linear but just due to endogeneity that OLS fails to estimate the true slope (I believe that).

If the nature generates data using some other model, then all the discussion above does not apply. But in practice, we never know what model nature used. So just fit the OLS, since you can always do that, then check conditional mean of the residuals from the OLS $E[\hat{e}_i|x]$, if it is roughly 0, great. If it is not, try to do predictor transformation (but it is hard, I don't know if there are standard guideline to perform this). There is also another issue, many cases, due to the high dimension of $x$, you don't have enough data for each value of the vector $x$, which means you can't estimate $E[\hat{e}_i|x_i]$ robustly (if you have enough data, then you probably will just estimate $E[y|x]$ non-parametrically and forget about OLS).


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.