# Regression and the CEF

"Regression offers a way of approximating the CEF linearly i.e,...,Thus, even if the CEF is non-linear as in the recipe and star rating example, the regression line would provide the best linear approximation to it (drawn in green below)".

I understand that we use conditional expectation in OLS assuming that the conditional expectation of Y for a ceratin value of X is true when Y can be modelled as a linear function of X. Please correct if I am misundestanding something.

So, and this can sound a silly (sorry!), what we are REALLY DOING when performing linear regression, is just aproximating the conditional expectation function? I mean, what we are really trying to estimate in a regression is the line that best fit the conditional expectation function? Or there is something I am no getting right. Please do not hesitate in correcting me :)

Oh, and one last thing If so, what is the necessity to do linear regression? Why we can't just work with the conditional expectation function instead of linear regression since is the conditional expectation function what we are trying to approxitamate to?

Regression and the CEF

The concept of regression can appear quite vast and sometimes ambiguous, especially if we refers on econometrics literature. However in its genuine sense, regression function and conditional expectation function (CEF) are simply synonym. In math terms:

$$E[y|X=x]=g(X)$$

is the regression of $$y$$ on $$X$$

where $$y$$ is a scalar rv (regressand) and $$X$$ a vector of rvs (regressors/covariates).

One problem is that, in general, in order to compute a precise form of $$g(X)$$ a completely specified joint distribution $$F(y,X)$$ is needed. In general this form can be not linear and not easy to compute and use. Also for these reasons the linear regression is much more used, frequently speaker talk about regression but have in mind linear regression. It is

$$E[y|X=x] = \beta_0 + \beta_1 x_1 + ... + \beta_{k}x_{k}$$

even if it is most common refer on the form that include the regression error:

$$y = \beta_0 + \beta_1 x_1 + ... + \beta_{k}x_{k} + v$$

Under some distributional restrictions, for example under joint Normality, regression and linear regression coincide. Otherwise linear regression give us a linear approximation of $$g(X)$$. It seems me that the most relevant advantage of linear regression is that can be computed even if the underlying joint distribution is unknown; truly some condition should be verified but that are quite weak. This discussion is related: Regression's population parameters

So

what we are REALLY DOING when performing linear regression, is just aproximating the conditional expectation function?

Exactly.

I mean, what we are really trying to estimate in a regression is the line that best fit the conditional expectation function?

This is precisely what OLS estimator do.

what is the necessity to do linear regression? Why we can't just work with the conditional expectation function instead of linear regression since is the conditional expectation function what we are trying to approxitamate to?

Simplification that come from the approximation argument discussed above is the main motivation. Other, strongly related, arguments deal with difficulties that we can encounter in estimation side for non linear regressions.

Note that the usefulness and scope of regression is another, very long, story. Read here can help: Regression: Causation vs Prediction vs Description

• Thanks for answering. Is the CEF function the same that the Population Regression Equation? Aug 27, 2020 at 18:29
• “Regression” is a word much more used in estimation that, let me say, in exact stat math theory. However CEF and regression can be intended as synonym. So, both can be intended as sample or population quantity. Terminologically the “regression equation” usually include the error also: $y = E[y|X] + \epsilon$. Frequently is used in population sense Aug 28, 2020 at 9:49
• What If our X is fixed, not random like econometrics books propose, is still CEF a synonym of regression? Because I understand that we can make regression both conditionaly and unconditionaly. I'm so sorry for making this silly questions man, but it is confussing that concept of conditional or unconditional when it comes to regression. Aug 28, 2020 at 14:06
• Concepts like joint probability distribution and CEF are properly defined only if also regressors (X) are random. However most of things you encounter in regression analysis take X are given. Then, consider X as constant or random, but given, is not so different. Aug 28, 2020 at 14:58
• Moreover "warning". Sometimes we can read something like "regressors fixed by an experimenter". Phrase like this reveal confusion between statistical and causal concepts. Them can bring readers in any sort of ambiguity. Read here: stats.stackexchange.com/questions/215230/… Nov 5, 2021 at 11:32

Oh, and one last thing If so, what is the necessity to do linear regression? Why we can't just work with the conditional expectation function instead of linear regression since is the conditional expectation function what we are trying to approxitamate to?

Here are three (good) reasons why linear regression is widely used to approximate the CEF.

1. Suppose $$X$$ has ten variables, and suppose we actually knew the true CEF, $$E[Y|X=x]=f(x)$$. How can we grasp the implications of this function? That's really hard in general. If the function were linear and additive in those 10 variables we will find this much easier. So we might want to summarise the function with its best linear and additive approximation for interpretation purposes even if we knew its true shape. Interpretation is really important in many applications because at least part of the credibility of an analysis rests on whether the modelled relationship is consistent with domain knowledge, in addition to statistical measures. So you need to understand the modelled relationship, and also have an idea what features in the data drive it. Incidentally, linear regression is a good basis from which to model departures from additivity and linearity: Interactions relax the former, variable transformations the latter. The modeller can easily incorporate these into the analysis. And more systematic approaches (eg regression splines) exist too.

2. We never know the true CEF in empirical work. We have to estimate it from data. We know from the theory of non-parametric regression that it takes an extraordinary amount of data to estimate such a function with any precision - if we really do not put any restrictions on it. An intuitive way of seeing this is to suppose you have 10 binary variables. Suppose you need about 50 data points for a particular value of $$x$$ (10 dimensional) to get a reasonable estimate of the average value of $$y$$. There are 1024 unique values of $$x$$ for which you need those 50 data points if you want to estimate the CEF at that level of precision for all values of $$x$$

3. Estimating a linear regression with OLS is computationally faster and more robust compared to just about any alternative. When you estimate a more complex model you often need to spend much more time on fitting it and on checking how numerically stable and robust the result is.