An image might help. In this image, we see a geometric view of the fitting.
Least squares finds a solution in a plane that has the closest distance to the observation.
(more general a higher dimensional plane for multiple regressors and a curved surface for non-linear regression)
In this case, the vector between observation and solution is perpendicular to the plane (a space spanned be the regressors), and perpendicular to the regressors.
Regularized regression finds a solution in a restricted set inside the the plane that has the closest distance to the observation.
In this case, the vector between observation and solution is not anymore perpendicular to te plane and not anymore perpendicular to the regressors.
But, there is still some sort of perpendicular relation, namely the vector of the residuals is in some sense perpendicular to the edge of the circle (or whatever other surface that is defined by te regularization)

The model of $\hat{y}$
Our model gives estimates of the observations, $\hat{y}$, the observations as function of parameters $\beta_i$.
$$\hat{y} = f(\beta)$$
In our image this is a linear function with two parameters $\beta_0$ and $\beta_1$
(you can of course generalize this to a large size of coefficients and observations, for simplicity we regard three observations and two coefficients such that we can plot it)
$$\begin{bmatrix} \hat{y}_{1} \\ \hat{y}_{2} \\ \hat{y}_{3} \end{bmatrix} = \beta_0 \begin{bmatrix} 1 \\ 1 \\ 1 \end{bmatrix}+
\beta_1 \begin{bmatrix} 0 \\ 1 \\ 2 \end{bmatrix}$$
The possible solutions of the model, defined by this linear sum, is represented by the red plane in the image.
Note that this plane in the image relates to the possible solutions of $y_i = \beta_0 + \beta_1 x_i$, when $[x_1,x_2,x_3] = [0,1,2]$. So we plotted the space of all possible $y_i$ (which is a 3D-space, and more generally a n-dimensional space) and the possible solutions that the model allows is a plane inside this space.
Finding the best model with least squares
The model allows any solution in the plane spanned by the model (in the image this is the 2D red plane, in general this can be a higher dimensional plane, and also it does not need to be linear).
The least-squares method will select the 'solution' $\hat{y} = \hat\beta_0 + \hat\beta_1 x_1 $ that has the lowest difference in terms of the squares of the residuals.
In geometric terms, this is equal to finding the point in the plane that has the smalles euclidian distance to the observed value. This smallest difference is achieved when the vector of residuals is orthogonal to the plane.
Finding the best model with ridge regression (or other regularization)
When we apply a penalty then this is similar to applying some constraint like 'the sum of vectors can not be above some value'. In the image this is represented by the purple drawing.
The solution is still inside the plane, but also inside the circle. Now the estimate solution is still a representing a shortest distance between the space of solutions and the observation. But the optimal solution is not anymore orthogonal projection onto the red plane. It is instead the shortes distance to the purple circles.