Edited with added material in response to comments by @whuber
This is an answer based on probability theory, not statistical estimates, so your mileage may vary.
If random variables $X$ and $Y$ have correlation coefficient $\rho$, then the linear least-mean-square error estimate of $Y$ given the value of $X$ is
$$\hat{Y} = \mu_Y + \rho\frac{\sigma_Y}{\sigma_X}(X - \mu_X),$$
and similarly,
the linear least-mean-square error estimate of $X$ given the value of $Y$ is
$$\hat{X} = \mu_X + \rho\frac{\sigma_X}{\sigma_Y}(Y - \mu_Y).$$
Note that $\hat{Y}$ and $\hat{X}$ are random variables that are linear functions of $X$ and $Y$ respectively. Their means are
$$\begin{align*}
\mu_{\hat{Y}} &=E[\hat{Y}] = E\left[\mu_Y+\rho\frac{\sigma_Y}{\sigma_X}(X - \mu_X)\right]
= \mu_Y+ \rho\frac{\sigma_Y}{\sigma_X}E[X - \mu_X] = \mu_Y\\
\mu_{\hat{X}} &= E[\hat{X}] = E\left[\mu_X+\rho\frac{\sigma_X}{\sigma_Y}(Y - \mu_Y)\right]
= \mu_Y+ \rho\frac{\sigma_X}{\sigma_Y}E[Y - \mu_Y] = \mu_X
\end{align*}$$
while the variances are
$$\begin{align*}
\sigma_{\hat{Y}}^2 &= E[(\hat{Y} - \mu_{\hat{Y}})^2]
= \frac{\rho^2\sigma_Y^2}{\sigma_X^2}E[(X-\mu_X)^2] = \rho^2\sigma_Y^2\\
\sigma_{\hat{X}}^2 &= E[(\hat{X} - \mu_{\hat{X}})^2]
= \frac{\rho^2\sigma_X^2}{\sigma_Y^2}E[(Y-\mu_Y)^2] = \rho^2\sigma_X^2
\end{align*}$$
Finally, the variances of the residual
errors $Y - \hat{Y}$ and $X - \hat{X}$ are $\sigma_Y^2(1-\rho^2)$ and
$\sigma_X^2(1-\rho^2)$ respectively. One can think of these results as
follows.
If we use the mean $\mu_Y$ as an estimate for $Y$, the mean-square error is $\sigma_Y^2$, but if we know the value of $X$ and use $\mu_Y + \rho\frac{\sigma_Y}{\sigma_X}(X - \mu_X)$ as the estimate of $Y$, the mean-square error is reduced
to $\sigma_Y^2(1-\rho^2)$.
If we use the mean $\mu_X$ as an estimate for $X$, the mean-square error is $\sigma_X^2$, but if we know the value of $Y$ and use $\mu_X + \rho\frac{\sigma_X}{\sigma_Y}(Y - \mu_Y)$ as the estimate of $X$, the mean-square error is reduced
to $\sigma_X^2(1-\rho^2)$.
In both cases, the mean-square error is reduced by the same fraction
$(1-\rho^2)$.
In terms of scatter plots (for discrete random variables or data)
on a plane with coordinate axes $x$ and $y$,
we have two straight lines
$$
\begin{align*}
y &= \mu_Y + \rho\frac{\sigma_Y}{\sigma_X}(x - \mu_X),\\
x &= \mu_X + \rho\frac{\sigma_X}{\sigma_Y}(y - \mu_Y),
\end{align*}
$$
of different slopes $\rho\sigma_Y/\sigma_X$ and
$\sigma_Y/\rho\sigma_X$ passing through the mean point
$(\mu_X,\mu_Y)$. The reason for the different slopes is that
we are choosing the slope to minimize the sum of the squares of
the vertical
distances of the points from the line in the first case, and to minimize the
sum of the squares of the horizontal distances of the points from the
line in the second case. These sums of squared distances are
$\sigma_Y^2(1-\rho^2)$ and $\sigma_X^2(1-\rho^2)$ respectively.
As a simple example, suppose that $(X,Y)$ takes on values $(0,0)$, $(0,1)$ and
$(1,1)$ with equal probability $\frac{1}{3}$ each or we have a scatter plot
with these three points. One can grind through the calculations if desired,
but it should be intuitively obvious that we should estimate $Y$ as $\frac{1}{2}$
if $X = 0$ and as $1$ if $X = 1$, while we should estimate $X$ as $0$ if
$Y = 0$ and as $\frac{1}{2}$ if $Y = 1$, that is the two lines have different
slopes $\frac{1}{2}$ and $2$ (in fact, reciprocal slopes since $\sigma_X^2 = \sigma_Y^2 = \frac{2}{9}$ in this example). This is what probability
theory gives. But if you treat the three points as a small sample from
from an unknown population and use estimates of the population
means, variances and correlation coefficent, then your results may be
different.
