How does ridge regression solve the multidimensionality problem if it doesn't assign zero to some coefficients

Question

I want to understand how does ridge regression solve the multidimensionality problem (when number of X variable is higher than the number of observations)? It shrinks the coefficients by introducing bias by incorporating the lambda term. It is clear that it will solve the overfitting issue. But how does it solve the multidimensionality.

Answer 1

I am assuming that the multidimensionality problem to be "solved" is the issue that when $p>n$, there will in general be many possible values of $\beta$ that all lead to equally good predictions of $Y$. In this case, I roughly interpret your question as asking "how does ridge adjudicate which of the many equally well fitting values of $\beta$ should we choose as our estimated $\beta$?"

To gain some intuition for what sort of implicit prior ridge will place on your coefficients, it is helpful to consider a very simple toy model. Suppose that $X_1 = X_2 = \cdots = X_K$, and the true model is of the form $$Y = \beta_0 + \beta_1 X_1 + \varepsilon$$ Suppose you wanted to fit a linear model $$Y = b_0 + b_1 X_1 + \cdots + b_K X_K + \varepsilon$$ Clearly, many different values of the $b_1,\ldots,b_K$ would produce the same predictions. In particular, any value of the parameters with $\sum_{k=1}^K b_k = \beta_1$ would give you the exact same predictions. So sample fit does not help distinguish between which of these models we should "prefer".

How then, does ridge solve this problem? For this very simple toy problem, we can think of this problem in two stages. First, given the artificial setup I have stated above, we note that the predictions of the model are the exact same whenever $\sum_{k=1}^K b_k = b$ for some constant $b$. Thus, the ridge optimization problem can be split up into two stages. First, pick the best slope $b$, and then fixing $b$, find the minimum sum-of-squared coefficients summing to $b$. $$\begin{aligned}\min_{b_1,\ldots,b_K} &\sum_{i=1}^N (Y_i - b_0 - \sum_{k=1}^K b_k X_k)^2 + \lambda\sum_{k=1}^K ||b_K||^2 = \\\min_{b} &\sum_{i=1}^N (Y_i - b_0 - b X_1)^2 + \lambda \min_{b_1,\ldots,b_K} \sum_{k=1}^K ||b_K||^2 \text{ s.t } \sum_{k=1}^K b_K = b\end{aligned}$$ For fixed $b$, the solution to the second optimization problem is simply to set $b_1=\cdots=b_K=\frac bK$.

This toy model highlights the basic high level way in which ridge will favor one model over another, even as their in-sample predictive performances are similar. If some of your $X$s basically have the same predictive content, ridge tends to prefer to spread out the coefficient evenly amongst all of your $X$s. In this way, even if there is no unambiguous data-driven "best" coefficient from the perspective of in-sample fit, ridge will choose the "best" amongst equally well-fitting models breaking ties through splitting the coefficients evenly amongst the various coefficients. If you do not think this is a reasonable way of breaking ties, that is fair. It means that your prior beliefs about what models are likely to describe the world probably disagree with the implicit prior assumption ridge is imposing.