The relation is not symmetric because we are solving two different optimisation problems. $\textbf{ Doing regression of $y$ given $x$}$ can be written as solving the following problem: $$\min_b \mathbb E(Y - bX)^2$$
whereas for $\textbf{doing regression of $x$ given $y$}$: $$\min_b \mathbb E(X - bY)^2$$, which can be rewritten as:
$$\min_b \frac{1}{b^2} \mathbb E(Y - bX)^2$$
It is also important to note that, two different-looking problems may have the same solution.