- If $\lambda \rightarrow \infty$ then our penalty term will be infinite for any $\beta$ other than $\beta = 0$, so that's the one we'll get. There is no other vector that will give us a finite value of the objective function.
(Update: Please see Glen_b's answer. This is not the correct historical reason!)
- This comes from ridge regression's solution in matrix notation. The solution turns out to be
$$
\hat \beta = (X^TX + \lambda I)^{-1} X^TY.
$$
The $\lambda I$ term adds a "ridge" to the main diagonal and guarantees that the resulting matrix is invertible. This means that, unlike OLS, we'll always get a solution.
Ridge regression is useful when the predictors are correlated. In this case OLS can give wild results with huge coefficients, but if they are penalized we can get much more reasonable results. In general a big advantage to ridge regression is that the solution always exists, as mentioned above. This applies even to the case where $n < p$, for which OLS cannot provide a (unique) solution.
Ridge regression also is the result when a normal prior is put on the $\beta$ vector.
Here's the Bayesian take on ridge regression:
Suppose our prior for $\beta$ is $\beta \sim N(0, \frac{\sigma^2}{\lambda}I_p)$. Then because $(Y|X, \beta) \sim N(X\beta, \sigma^2 I_n)$ [by assumption] we have that
$$
\pi(\beta | y) \propto \pi(\beta) f(y|\beta)
$$
$$
\propto \frac{1}{(\sigma^2/\lambda)^{p/2}} \exp \left( -{\lambda \over 2\sigma^2} \beta^T\beta \right) \times \frac{1}{(\sigma^2)^{n/2}} \exp \left( \frac{-1}{2\sigma^2} ||y - X\beta||^2 \right)
$$
$$
\propto \exp \left( -{\lambda \over 2\sigma^2} \beta^T\beta - \frac{1}{2\sigma^2} ||y - X\beta||^2 \right).
$$
Let's find the posterior mode (we could look at posterior mean or other things too but for this let's look at the mode, i.e. the most probable value).
This means we want
$$
\max_{\beta \in \mathbb R^p} \ \exp \left( -{\lambda \over 2\sigma^2} \beta^T\beta - \frac{1}{2\sigma^2} ||y - X\beta||^2 \right)
$$
which is equivalent to
$$
\max_{\beta \in \mathbb R^p} \ -{\lambda \over 2\sigma^2} \beta^T\beta - \frac{1}{2\sigma^2} ||y - X\beta||^2
$$
because $\log$ is strictly monotone and this in turn is equivalent to
$$
\min_{\beta \in \mathbb R^p} ||y - X\beta||^2 + \lambda \beta^T\beta
$$
which ought to look pretty familiar.
Thus we see that if we put a normal prior with mean 0 and variance $\frac{\sigma^2}{\lambda}$ on our $\beta$ vector, the value of $\beta$ which maximizes the posterior is the ridge estimator. Note that this treats $\sigma^2$ more as a frequentist parameter because there's no prior on it but it isn't known, so this isn't fully Bayesian.
Edit: you asked about the case where $n < p$.
We know that a hyperplane in $\mathbb R^p$ is defined by exactly $p$ points. If we are running a linear regression and $n = p$ then we exactly interpolate our data and get $||y - X\hat\beta||^2 = 0$. This is a solution, but it is a terrible one: our performance on future data will most likely be abysmal. Now suppose $n < p$: there is no longer a unique hyperplane defined by these points. We can fit a multitude of hyperplanes, each with 0 residual sum of squares.
A very simple example: suppose $n = p = 2$. Then we'll just get a line between these two points. Now suppose $n = 2$ but $p = 3$. Picture a plane with these two points in it. We can rotate this plane without changing the fact that these two points are in it, so there are uncountably many models all with a perfect value of our objective function, so even beyond the issue of overfitting it is not clear which one to pick.
As a final comment (per @gung's suggestion), the LASSO (using an $L_1$ penalty) is commonly used for high dimensional problems because it automatically performs variable selection (sets some $\beta_j = 0$). Delightfully enough, it turns out that the LASSO is equivalent to finding the posterior mode when using a double exponential (aka Laplace) prior on the $\beta$ vector. The LASSO also has some limitations, such as saturating at $n$ predictors and not necessarily handling groups of correlated predictors in an ideal fashion, so the elastic net (convex combination of $L_1$ and $L_2$ penalties) may be brought to bear.
