#A geometrical interpretation
The estimator described in the question is the Lagrange multiplier equivalent of the following optimization problem:
$$\text{minimize $f(\beta)$ subject to $g(\beta) \leq t$ and $h(\beta) = 1$ } $$
$$\begin{align}
f(\beta) &= \lVert y-X\beta \lVert^2 \\
g(\beta) &= \lVert \beta \lVert^2\\
h(\beta) &= \lVert X\beta \lVert^2
\end{align}$$
normal
which can be viewed, geometrically, as finding the smallest ellipsoid $f(\beta)=RSS$ that touches the intersection of the sphere $g(\beta) = t$ and the ellipsoid $h(\beta)=1$
----------
## Comparison to the standard ridge regression view
In terms of a geometrical view this changes the *old* view (for standard ridge regression) of the point where **a spheroid (errors) and sphere ($\|\beta\|^2=t$) touch**. Into a new view where we look for the point where **the spheroid (errors) touches a curve (norm of beta constrained by $\|X\beta\|^2=1$)**. The one sphere (blue in the left image) changes into a lower dimension figure due to the intersection with the $\|X\beta\|=1$ constraint.
In the two dimensional case this is simple to view.
[![geometric view][1]][1]
When we tune the parameter $t$ then we change the relative length of the blue/red spheres or the relative sizes of $f(\beta)$ and $g(\beta)$ *(In the theory of Lagrangian multipliers there is probably a neat way to formally and exactly describe that this means that for each $t$ as function of $\lambda$, or reversed, is a monotonous function. But I imagine that you can see intuitively that the sum of squared residuals only increases when we decrease $||\beta||$.)*
The solution $\beta_\lambda$ for $\lambda=0$ is as you argued on a line between 0 and $\beta_{LS}$
The solution $\beta_\lambda$ for $\lambda \to \infty$ is (indeed as you commented) in the loadings of the first principal component. This is the point where $\lVert \beta \rVert^2$ is the smallest for $\lVert \beta X \rVert^2 = 1$. It is the point where the circle $\lVert \beta \rVert^2=t$ touches the ellipse $|X\beta|=1$ in a single point.
In this 2-d view the edges of the intersection of the sphere $\lVert \beta \rVert^2 =t$ and spheroid $\lVert \beta X \rVert^2 = 1$ are points. In multiple dimensions these will be curves
*(I imagined first that these curves would be ellipses but they are more complicated. You could imagine the ellipsoid $\lVert X \beta \rVert^2 = 1$ being intersected by the ball $\lVert \beta \rVert^2 \leq t$ as some sort of ellipsoid frustum but with edges that are not a simple ellipses)*
-----
##Regarding the limit $\lambda \to \infty$
*At first (previous edits) I wrote that there will be some limiting $\lambda_{lim}$ above which all the solutions are the same (and they reside in the point $\beta^*_\infty$). But this is **not** the case*
Consider the optimization as a LARS algorithm or gradient descent. If for any point $\beta$ there is a direction in which we can change the $\beta$ such that the penalty term $|\beta|^2$ increases less than the SSR term $|y-X\beta|^2$ decreases then you are not in a minimum.
- In **normal ridge regression** you have a zero slope (in all directions) for $|\beta|^2$ in the point $\beta=0$. So for all finite $\lambda$ the solution can not be $\beta = 0$ (since an infinitesimal step can be made to reduce the sum of squared residuals without increasing the
penalty).
- **For LASSO** this is *not* the same since: the penalty is $\lvert \beta \rvert_1$ (so it is not quadratic with zero slope). Because of that LASSO will have some limiting value $\lambda_{lim}$ above which all the solutions are zero because the penalty term (multiplied by $\lambda$) will increase more than the residual sum of squares decreases.
- **For the constrained ridge** you get the same as the regular ridge regression. If you change the $\beta$ starting from the $\beta^*_\infty$ then this change will be *perpendicular* to $\beta$ (the $\beta^*_\infty$ is perpendicular to the surface of the ellipse $|X\beta|=1$) and $\beta$ can be changed by an infinitesimal step without changing the penalty term but decreasing the sum of squared residuals. Thus for any finite $\lambda$ the point $\beta^*_\infty$ can not be the solution.
-----
> ##Further notes regarding the limit $\lambda \to \infty$
>
> - The usual ridge regression limit for $\lambda$ to infinity corresponds to a different point in the constrained ridge regression.
> This 'old' limit corresponds to the point where $\mu$ is equal to -1.
> Then the derivative of the Lagrange function in the normalized problem
>
> $$2 (1+\mu) X^{T}X \beta + 2 X^T y + 2 \lambda \beta$$ corresponds
> to a solution for the derivative of the Lagrange function in the
> standard problem
>
> $$2 X^{T}X \beta^\prime + 2 X^T y + 2 \frac{\lambda}{(1+\mu)}
\beta^\prime \qquad \text{with $\beta^\prime = (1+\mu)\beta$}$$
>
> - You could parameterize the 2-d problem as following:
>
> $$\beta(\theta) = \begin{pmatrix} a \cos(\theta) \\ b
\sin(\theta) \end{pmatrix} $$
>
> e.g. the regression is with:
>
> $$ X = \begin{pmatrix} 1/a & 0 \\ 0 & 1/b \\ \end{pmatrix}$$
>
>
> Then the following is to be minimized $$f(\theta) =
\left(\cos(\theta)-\frac{\hat{\beta}_{LS}}{a}\right)^2 +
\left(\sin(\theta)-\frac{\hat{\beta}_{LS}}{b}\right)^2 + \lambda
\left( a^2 \cos(\theta)^2 + b^2 \sin(\theta)^2 \right) $$ It is a
> bit more work to simplify those trigonometric functions, but note that
> the part $$\left( a^2 \cos(\theta)^2 + b^2 \sin(\theta)^2 \right)$$
> has slope zero in the points $\theta = \frac{k}{2}\pi$, but the part
> $$\left(\cos(\theta)-\frac{\hat{\beta}_{LS}}{a}\right)^2 +
\left(\sin(\theta)-\frac{\hat{\beta}_{LS}}{b}\right)^2 $$ does not. So
> no matter what the value of $\lambda$ the solution does not have a
> minimum in $\theta = \frac{k}{2}\pi$, although we can get arbitrarily
> close when $\lambda \to \infty$.
[1]: https://i.sstatic.net/XwPj6.png
[2]: https://i.sstatic.net/QrueJ.gif
[3]: https://i.sstatic.net/hNgns.png