Why is the conditional expectation the best predictor but only if we have the joint distribution?

Question

If we want to predict one variable $Y$ based on another $X$, the best predictor is apparently $\mathbb{E}[Y \mid X = x]$. However, this apparently assumes two things:

1. The distribution is symmetric.

   Which distribution needs to be symmetric? I think it is the conditional one, that is, $P(Y \mid X=x)$, because the mean would be the best predictor only if the distribution is symmetric. If it wasn't symmetric, then maybe the median would be a better predictor.

2. We need to know the joint distribution of $Y$ and $X$, that is, we need to know $P(X=x, Y=y)$.

   Why do we strictly need to know $P(X=x, Y=y)$? I understand that, if we have the joint distribution, e.g. a table of all combinations of values of $X$ and $Y$, we can filter out all $Y$ values where $X \neq x$ and get only those where $X = x$, then we take the mean of those. Anyway, this should be the idea, which is applied in practice. But $\mathbb{E}[Y \mid X = x]$ should still be the best predictor (assuming symmetry), either we have the joint distribution or not.

Answer 1

In this context, the best predictor is the predictor that minimizes the variance $E[(Y-g(X))^2]$, where $g(X)$ is the prediction of $Y$, given the data $X$. Several analyses show that this expectation is minimized when $g(X)=E[Y|X]$. So, no symmetric assumption over any of the PDFs (PMFs). By the way, in general, you need to know $p_{Y|X}(y|x)$ to calculate the expected value. You can deduce it from the joint distribution, but having the complete joint is not necessary.

Median, or something else might sometimes result in a better estimate, but better in other sense. Not the variance defined above.