The relation is not symmetric because we are solving two different optimisation problems. $\textbf{ Doing regression of $y$ given $x$}$ can be written as solving the following problem: $$\min_b \mathbb E(Y - bX)^2$$

whereas for $\textbf{doing regression of $x$ given $y$}$: $$\min_b \mathbb E(X - bY)^2$$, which can be rewritten as:

$$\min_b \frac{1}{b^2} \mathbb E(Y - bX)^2$$

The relation is not symmetric because we are solving two different optimisation problems. $\textbf{ Doing regression of $y$ given $x$}$ can be written as solving the following problem: $$\min_b \mathbb E(Y - bX)^2$$

whereas for $\textbf{doing regression of $x$ given $y$}$: $$\min_b \mathbb E(X - bY)^2$$, which can be rewritten as:

$$\min_b \frac{1}{b^2} \mathbb E(Y - bX)^2$$

It is also important to note that, two different-looking problems may have the same solution.