I'm trying to learn machine learning and I'm filling in the gaps in my knowledge as I go along. I see from this definition that

$$ E[X] = \int_{\mathbb{R}} xf(x) dx $$

But what is $E[\hat{\beta}|X]$? It is defined here. Is it expected value of $\hat{\beta}$ given $X$?

  • 3
    $\begingroup$ It is a conditional expectation. Heuristically, the value you expect $\hat\beta$ to take when you know $X$. Non heuristically, you can also study measure theory to learn about conditional expectation rigorously...might take a few years though. $\endgroup$ – Christoph Hanck Oct 5 at 10:16

Yes. it is the expected value of the estimator $\hat\beta$ of $\beta$ given your data $X$.


For example, consider the model $y = X\beta+\epsilon$ with $E[\epsilon|X] = 0$.

The ordinary least squares estimator of $\beta$ is $\hat{\beta} = (X'X)^{-1}X'y$, therefore you can calculate its conditional expectation as

$$E[\hat{\beta}|X] = E[(X'X)^{-1}X'y|X] = (X'X)^{-1}X'E[y|X] = \beta$$

This happens because a random variable $X$ conditioned on itself is a constant and you can take it out of the expectation.

| cite | improve this answer | |
  • $\begingroup$ Thanks. I have a follow up question. How may one estimate/calculate $E[\hat{\beta}|X]$ if one is given $X$ and corresponding observations $y$? $\endgroup$ – Nachiket Oct 5 at 10:35
  • $\begingroup$ I can see how one can calculate $\hat{\beta}$ using $\hat{\beta} = (X^TX)^{-1}X^Ty$ but can't see how to calculate $E[\hat{\beta}|X]$ $\endgroup$ – Nachiket Oct 5 at 10:43
  • $\begingroup$ Do you get now how to calculate it? $\endgroup$ – Ale Oct 5 at 10:50
  • $\begingroup$ No, I don't get it. I'm guessing that $E[y|X]=X\beta$ and therefore the math works out. But I cannot make sense of $E[\hat{\beta}|X]=\beta$. I would have expected $E[\hat{\beta}|X]$ to be a particular vector, not the original variable ($\beta$) itself. Could you point be to a very basic text on this? My background is in engineering and solid mechanics. $\endgroup$ – Nachiket Oct 5 at 11:24
  • $\begingroup$ $E[y|X] = E[X\beta+\epsilon|X] = X\beta$ as stated by assumption of $E[\epsilon|X] = 0$. The rest is linear algebra. $\endgroup$ – Ale Oct 5 at 11:34

Not the answer you're looking for? Browse other questions tagged or ask your own question.