Let me define my terms. Suppose I have a pair of jointly distributed random variables Y, X, where Y is numeric and X is a random vector. Note that I do not want to assume that Y and X are related in the usual sense of linear regression. Said a different way, I do not want to assume that the linear regression model is correct.

Let's say the true linear regressor (possibly "best linear regressor" would be better terminology?) is found my minimizing the expected square error over the joint distribution:

$$ \newcommand{\argmin}{\rm argmin} \argmin_{\beta} E_{X,Y} \left[ \left( Y - \overrightarrow{X} \beta \right)^2 \right] $$

And let's say the average linear regressor is found by taking the expectation of the linear regression solution with respect to a sampling distribution (where samples are of some fixed size) of X,Y:

$$ E_D \left[ \left( \mathbf{X}^t \mathbf{X} \right)^{-1} \mathbf{X}^t \overrightarrow{Y} \right] $$

here my samples D are stacked into matrices (for X) or a vector (for Y) in the usual way. The formula is the standard expression for the parameter vector given a sample data set.

My intuition tells me that these quantities are equal, but my attempts to prove it fail. I'm leaning towards this being false at this point due to the following computation:

$$ \begin{align} \argmin_{\beta} & E_{X,Y} \left[ \left( Y - \overrightarrow{X} \beta \right)^2 \right] \\ &= \argmin_{\beta} E_D \left[ \sum_i \left(Y_i - \overrightarrow{X_i} \beta \right)^2 \right] \\ &= \argmin_{\beta} E_D \left[ \left( \overrightarrow{Y} - \mathbf{X} \beta \right)^t \left( \overrightarrow{Y} - \mathbf{X} \beta \right) \right] \\ &= {\rm argzero}_{\beta} E_D \left[ \mathbf{X}^t \left( \overrightarrow{Y} - \mathbf{X} \beta \right)\right] \\ &\Rightarrow \beta = E_D \left[ \left( \mathbf{X}^t \mathbf{X} \right) \right]^{-1} E_D \left[ \mathbf{X}^t \overrightarrow{Y} \right] \end{align} $$

The issue is clearly that the product of expectations at the end has no reason to collapse to a single expectation.

I suppose my questions are simply:

  • Is my mathematics correct?
  • Why is my intuition flawed? Is there an intuitive way to see the difference, or a simple example where the two quantities are obviously different?

There was a question of exactly what I'm up to here, so let me explain how I got to this question.

Let's consider a simple toy model for definiteness:

$$ \begin{align} X \sim U(0, 2 \pi) \\ Y \sim \sin(X) + N(0, \epsilon) \end{align} $$

This could really be anything, I just want a very simple example. Let's say I have a learning algorithm, linear regression for definiteness:

$$ \mathcal{A} : D \mapsto f(X) $$

so $\mathcal{A}(D)$ is the linear regression of $X$ on $Y$ using the data set $D$ (which comes from the sampling distribution of $X, Y$). I want to calculate the model bias, and to do so I need the expected model, where the expectation is taken over the sampling distribution. The question is, how can I calculate this analytically? My intuition was to solve the optimization problem:

$$ \argmin_{\beta} E_{X,Y} \left[ \left( Y - \overrightarrow{X} \beta \right)^2 \right] $$

Which if you work out the math in this case, reduces to:

$$ \argmin_{a,b} \int_0^{2 \pi} \int_{-\infty}^{\infty} (y - ax - b)^2 e^{ \frac{(y - \sin(x))^2}{2 \epsilon^2}} dy dx $$

I can solve this numerically (using scipy), which gives the blue line below:

True Linear Regressor vs. Signal

Attempting to convince myself that the solution to this problem, the true linear regressor, is the same as what is called for in the definition of model bias, the average linear regressor, is where this problem came from.

  • $\begingroup$ What happens if you restrict the domain to $(0,\pi)?$ $\endgroup$
    – Dave
    Commented Sep 18, 2021 at 5:59

1 Answer 1


I am not sure what you are doing here, so only a quick comment. In general, for two random variables $Z$ and $W$, $E(ZW)\neq E(Z)E(W)$. I am not sure what your operator $E_D$ does but my guess it is that this property applies to $E_D$ too. So, the last line does not follows from the previous one.

  • $\begingroup$ That's what I meant at the end "The product of expectations has no reason to collapse to a single expectation." The $E_D$ operator is the expectation over a sampling distribution (samples from $X,Y$). The last line does follow from the previous one. Im only using that the expectation of a sum is the sum of the expectations. $\endgroup$ Commented Apr 25, 2015 at 16:14
  • $\begingroup$ After I posted this I found that my last equation (the product of expectations) appears in Hastie, Tibshirani and Friedman, chapter 2. It's equation 2.16 in the 10th printing. $\endgroup$ Commented Apr 25, 2015 at 16:19
  • $\begingroup$ We do not assume that the X's are random, but that they are fixed. Therefore, the expectation of X is X. $\endgroup$
    – ChinG
    Commented Nov 30, 2015 at 21:33
  • $\begingroup$ @ChinG No. You can see that all of my expectations are over either the joint distribution of X and Y, or over the sampling distribution of training data sets $D$. I am explicitly interested in the variation in X. $\endgroup$ Commented Dec 10, 2016 at 22:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.