I'm stuck on some subtleties involved in simple linear regression formulation.

In a very general fashion I understood that the optimal point forecast for a random variable $Y$ (in terms of MSE minimization) is the expected value, say : $$ y_{\operatorname{best}}=E[Y]$$

Knowing that $ Y $ is joined with another variable $X$, without information on any distribution (neither the joint nor the marginals), we can say that if we see a realization of $X$ say $X=x$ we can use this information to get a better optimal point forecast in respect to the simple expected value and it's the conditional one : $$ y_{\operatorname{best}}|_{X=x} = E[Y|X]$$

Assuming $x$ as a variable and not as a fixed point is a function of $x$. If we want this function to be linear (it's an imposition not an hp. on the true model), the best linear approximation we can make in term of MSE minimization is the one that has: $$\beta=\frac{\operatorname{cov}(X,Y)}{\operatorname{var}(X)} \qquad \alpha = E[Y]-\beta E[X].$$

Here I didn't assume that $X$ is fixed (non stochastic), I didn't assume additive noise etc. None of these assumption has been necessary to have $\beta$ and $\alpha$.

$\; $

$ \quad $ The problem on which I am stuck starts here :

if we want some good properties on $\alpha $ and $ \beta$ , if we want to check bias, constistency, variance etc. we actually need to make some non trivial asssumption. Typically in texts is assumed : $ Y=\alpha+\beta X+\epsilon$ on which we observe $y_i=\alpha +\beta x_i +\epsilon _i$. Then usually they substitute it in $\beta$ $$ \beta =\frac{\operatorname{cov}(X,\alpha + \beta X + \epsilon )}{\operatorname{var}(X)}$$ and we go further doing some expectation etc, making more assumption again to get desired properties (as uncorrelation between $X$ and $\epsilon$ to avoid bias).

1) Actually the model assumption seems to me not completely indipendent from the fact that we impose $E[Y|X]$ to be linear, so $E[Y|X]=\alpha +\beta x $. This suggest that $Y|X$ should only be something like $Y|X =\alpha +\beta x +\epsilon (x) $ with $ \epsilon (x) $ having zero mean. But how to get from $ Y|X = $ to $ Y = $ ? Maybe it's a dumb question but what's the matematical analogue of 'de-conditioning'a variable? And if there's not is legitimate to substitute $Y|X$ in the expression of $\beta$ instead of $Y$ ?

2) When I think to a conditional expectation I think to the fact (maybe i'm wrong) that the $X$ variable actually loose his stochasticity. But in the final result of $\beta$ it appears again having stochastic behaviour (there is a covariance and a variance terms). How can I interpretate right the 'conditioning' ? Is $E[Y|X]=\alpha + \beta x$ a simple function of real variable or it's $E[Y|X]=\alpha + \beta X$ actually a random variable?


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.