Regression and the CEF

Question

I recently read in this page (https://www.timlrx.com/2018/02/26/notes-on-regression-approximation-of-the-conditional-expectation-function/#fn1) that:

"Regression offers a way of approximating the CEF linearly i.e,...,Thus, even if the CEF is non-linear as in the recipe and star rating example, the regression line would provide the best linear approximation to it (drawn in green below)".

I understand that we use conditional expectation in OLS assuming that the conditional expectation of Y for a ceratin value of X is true when Y can be modelled as a linear function of X. Please correct if I am misundestanding something.

So, and this can sound a silly (sorry!), what we are REALLY DOING when performing linear regression, is just aproximating the conditional expectation function? I mean, what we are really trying to estimate in a regression is the line that best fit the conditional expectation function? Or there is something I am no getting right. Please do not hesitate in correcting me :)

Oh, and one last thing If so, what is the necessity to do linear regression? Why we can't just work with the conditional expectation function instead of linear regression since is the conditional expectation function what we are trying to approxitamate to?

Answer 1

Regression and the CEF

The concept of regression can appear quite vast and sometimes ambiguous, especially if we refers on econometrics literature. However in its genuine sense, regression function and conditional expectation function (CEF) are simply synonym. In math terms:

$E[y|X=x]=g(X)$

is the regression of $y$ on $X$

where $y$ is a scalar rv (regressand) and $X$ a vector of rvs (regressors/covariates).

One problem is that, in general, in order to compute a precise form of $g(X)$ a completely specified joint distribution $F(y,X)$ is needed. In general this form can be not linear and not easy to compute and use. Also for these reasons the linear regression is much more used, frequently speaker talk about regression but have in mind linear regression. It is

$$ E[y|X=x] = \beta_0 + \beta_1 x_1 + ... + \beta_{k}x_{k}$$

even if it is most common refer on the form that include the regression error:

$$ y = \beta_0 + \beta_1 x_1 + ... + \beta_{k}x_{k} + v $$

Under some distributional restrictions, for example under joint Normality, regression and linear regression coincide. Otherwise linear regression give us a linear approximation of $g(X)$. It seems me that the most relevant advantage of linear regression is that can be computed even if the underlying joint distribution is unknown; truly some condition should be verified but that are quite weak. This discussion is related: Regression's population parameters

So

what we are REALLY DOING when performing linear regression, is just aproximating the conditional expectation function?

Exactly.

I mean, what we are really trying to estimate in a regression is the line that best fit the conditional expectation function?

This is precisely what OLS estimator do.

what is the necessity to do linear regression? Why we can't just work with the conditional expectation function instead of linear regression since is the conditional expectation function what we are trying to approxitamate to?

Simplification that come from the approximation argument discussed above is the main motivation. Other, strongly related, arguments deal with difficulties that we can encounter in estimation side for non linear regressions.

Note that the usefulness and scope of regression is another, very long, story. Read here can help: Regression: Causation vs Prediction vs Description

Answer 2

Oh, and one last thing If so, what is the necessity to do linear regression? Why we can't just work with the conditional expectation function instead of linear regression since is the conditional expectation function what we are trying to approxitamate to?

Here are three (good) reasons why linear regression is widely used to approximate the CEF.

1. Suppose $X$ has ten variables, and suppose we actually knew the true CEF, $E[Y|X=x]=f(x)$. How can we grasp the implications of this function? That's really hard in general. If the function were linear and additive in those 10 variables we will find this much easier. So we might want to summarise the function with its best linear and additive approximation for interpretation purposes even if we knew its true shape. Interpretation is really important in many applications because at least part of the credibility of an analysis rests on whether the modelled relationship is consistent with domain knowledge, in addition to statistical measures. So you need to understand the modelled relationship, and also have an idea what features in the data drive it. Incidentally, linear regression is a good basis from which to model departures from additivity and linearity: Interactions relax the former, variable transformations the latter. The modeller can easily incorporate these into the analysis. And more systematic approaches (eg regression splines) exist too.

2. We never know the true CEF in empirical work. We have to estimate it from data. We know from the theory of non-parametric regression that it takes an extraordinary amount of data to estimate such a function with any precision - if we really do not put any restrictions on it. An intuitive way of seeing this is to suppose you have 10 binary variables. Suppose you need about 50 data points for a particular value of $x$ (10 dimensional) to get a reasonable estimate of the average value of $y$. There are 1024 unique values of $x$ for which you need those 50 data points if you want to estimate the CEF at that level of precision for all values of $x$

3. Estimating a linear regression with OLS is computationally faster and more robust compared to just about any alternative. When you estimate a more complex model you often need to spend much more time on fitting it and on checking how numerically stable and robust the result is.