# In some sense, is linear regression an estimate of an estimate of an estimate?

Consider the problem of estimating a random variable $$Y$$ using another random variable $$X$$.

The best estimator of $$Y$$ by a function of $$X$$ is the conditional expectation $$E[Y|X]$$. It minimizes the mean-squared error $$E[(Y - h(X))^2]$$ over all $$L^2$$-functions $$h(X)$$ of $$X$$.

However, $$E[Y|X]$$ seldom has a nice closed expression. A further approximation is to consider the best linear unbiased estimator $$linear[Y|X] = \mu_Y + \rho \frac{\sigma_Y}{\sigma_X} (X - \mu_X)$$

However, in practice we don't even know the population variances and population correlation. So we use linear regression techniques such as the least-squared solution to get an estimate from a random sample.

In this sense, can we say that linear regression is an estimate of an estimate ($$linear[Y|X]$$) of an estimate ($$E[Y|X]$$) of the original random variable ($$Y$$)?

To some extent, you had some very good point. The biggest problem in your interpretation is that you confused the concepts of approximation and estimation.

By probability theory, there exists a Borel function $$f: \mathbb{R} \to \mathbb{R}$$ such that $$E[Y|X] = f(X)$$ (almost surely). As you stated, for general distribution of $$(X, Y)$$, $$f$$ seldom has a nice closed form. On the other hand, suppose by some means, we have collected $$n$$ observations $$S = \{(x_1, y_1), \ldots, (x_n, y_n)\}$$ from the underlying distribution $$(X, Y)$$. A natural problem in statistics is then: can we use the sample $$S$$ to make some inference on the unknown $$f$$? Note that, while it is standard to say "estimate the functional form of $$f$$ using $$S$$", it is conceptually incorrect to say "estimate the random variable $$Y$$ using another random variable $$X$$", for the following two reasons:

1. No random variable $$Y$$ can be "estimated" by another random variable $$X$$. This has been refuted in Tim's answer. I just want to add that, if you recall that a random variable is essentially a real-valued function, does it make much sense to say use one function to "estimate" another function? From probability perspective, the statement "$$E(Y|X)$$ minimizes the mean-squared error $$E[(Y - h(X))^2]$$ over all $$L^2$$-functions $$h(X)$$ of $$X$$" is good enough but needs some correction as well: the "mean-squared error" has to be the "conditional mean-squared error" $$E[(Y - h(X))^2|X]$$. Do not use "estimator" for $$E(Y|X)$$ because $$Y$$ is not a valid estimand from statistics perspective (also see elaborations below).
2. In statistics (at least in frequentist statistical inference), the terminology "estimation" specifically means using an observed sample to draw some information on some unknown, yet non-random quantities (called "parameters") of an underlying population (or equivalently, distribution). From this perspective, your misuse of the word "estimate" is obvious: throughout your question, there is only one place you mentioned "sample": "So we use linear regression techniques such as the least-squared solution to get an estimate from a random sample." To be fair, this is the only place that you used the word "estimate" correctly, whereas "estimate/estimator" appeared in other places do not align with their standard statistical usages.

The more appropriate word for the problem you described is "approximation" (you actually also mentioned this term once but for the most time confused it with "estimation"): since $$E[Y|X] = f(X)$$ in general does not have an analytical form (i.e., "probability deduction" failed to work here), we need to turn to the help of statistical inference. But in order to get the statistics machine running, the first question we need to face is: what statistical tools should we use? Parametric inference or non-parametric inference? It turns out that the linear model is the simplest parametric inference weapon that practitioners like to use, which means, you specify $$f(X)$$ (it may well be a completely wrong specification, but the advantage is its simplicity and interpretability) as a linear function of $$X$$, i.e., $$f(X) = \alpha + \beta X$$, and then go head and use $$S$$ to estimate the parameters $$\alpha$$ and $$\beta$$. The procedure of specifying the unknown functional form $$f$$ as a linear function $$\alpha + \beta X$$ with just two unknown parameters is approximation (or model specification), it is not estimation, which is actually the next step that follows model specification. It is clear that while estimation cannot be done without sample/data, approximation in principle can be done without data (because it is just about selecting a simpler function to proxy a complicated function). However, to make a satisfactory approximation (i.e., build a decent model) requires the guidance of data as well and is usually interweaved with estimation in an iterative style. In this sense, "approximation" and "estimation" are closely related.

It is worth mentioning that when the joint distribution of $$(X, Y)$$ is bivariate Gaussian, then approximating $$f$$ by $$\alpha + \beta X$$ becomes exact. However, this doesn't make $$\alpha + \beta X$$ the best linear unbiased estimator of $$f(X)$$ when $$(X, Y)$$ is non-Gaussian. The "best linear unbiased estimator" refers to the estimator $$(\hat{\alpha}, \hat{\beta})$$ that minimizes variance after you have approximated $$f(X)$$ by $$\alpha + \beta X$$. It is well known that when the error distribution is spherical, the best linear unbiased estimator is the ordinary least-squares estimator.

Finally, let me quote the opening remark of Chapter 5 in The Elements of Statistical Learning to consolidate the point made above. If you want to get a better, more realistic approximation to $$f$$ than linear model, you can start looking into this chapter too.

We have already made use of models linear in the input features, both for regression and classification. Linear regression, linear discriminant analysis, logistic regression and separating hyperplanes all rely on a linear model. It is extremely unlikely that the true function $$f(X)$$ is actually linear in $$X$$. In regression problems, $$f(X) = E(Y |X)$$ will typically be nonlinear and nonadditive in $$X$$, and representing $$f(X)$$ by a linear model is usually a convenient, and sometimes a necessary, approximation. Convenient because a linear model is easy to interpret, and is the first-order Taylor approximation to $$f(X)$$.

It's not.

Consider the problem of estimating a random variable $$Y$$ using another random variable $$X$$.

We don't estimate random variables, but things about random variables. What would it mean to “estimate the random variable”? If I told you that I “estimated that the length of life is 70 years” what would it mean? Everybody would live exactly 70 years? It's maximum, minimum, average, mode, or median..? The statement would be quite meaningless. Also, why would you “estimate” a random variable with a single value when you can estimate its distribution (e.g. with empirical distribution or kernel density)?

The best estimator of $$Y$$ by a function of $$X$$ is the conditional expectation $$E[Y|X]$$. [...]

$$E[Y]$$ or $$E[Y|X]$$ are not estimators, but properties of random variables. The estimator is a function of a sample, the expected value is a property of a random variable.

However, $$E[Y|X]$$ seldom has a nice closed expression. [...] So we use linear regression techniques such as the least-squared solution to get an estimate from a random sample.

One has nothing to do with the other. To find $$E[Y]$$ you need to solve an integral. $$E[Y]$$ applies to mathematical objects (random variables). Linear regression needs data, if I asked you to tell me what is the expected value given that you know the probability density function of the variable, you wouldn't be able to calculate linear regression on the given function. There wouldn't be a closed-form solution either. The opposite is also true, you cannot calculate the expected value from the data, but you can use some estimator to approximate it.

A frequentist would use the term "prediction" for what you call "estimation of a random variable". Furthermore, if you don't assume a linear relationship but just use this as approximation, using the term "estmation" for "approximation" is confusing. And even then what is "estimated" is the coefficients of the regression, not the prediction of $$Y$$. So you're estimating the coefficients of an approximation in order to predict $$Y$$.

Things are not so clear in Bayesian analysis where parameters to be estimated are also random variables. There $$E[Y|X]$$ may be legitimately called "estimator" (even though some would probably still see this as confusing use of terminology), and you may estimate involved regression parameters on top of it. I don't see any way how to justify the term "estimator" for the linear approximation though in a Bayesian framework.