Return to Question

In Introduction to Statistical Learning, in the part where ridge regression is explained, the authors say that

As $\lambda$ increases, the flexibility of the ridge regression fit decreases, leading to decreased variance but increased bias.

Here is my take on proving this line:
In ridge regression we have to minimize the sum:$$RSS+\lambda\sum_{i=0}^n\beta_i\\=\sum_{i=1}^n(y_i-\beta_0-\sum_{j=1}^p\beta_jx_{ij})^2+\lambda\sum_{j=1}^p\beta_j^2$$
Here, we can see that a general increase in the $\beta$ vector will decrease $RSS$ and increase the other term. So, in order to minimize the whole term, a kind of equilibrium must be made between the $RSS$ term and the $\lambda\sum_{i=0}^p\beta_i$ term. Let their sum be $S$.
Now, if we increase $\lambda$ by $1$, then by using the previous value of the $\beta$ vector, $\lambda\sum_{i=1}^p\beta_i$ will increase, whereas $RSS$ will remain the same. Thus $S$ will increase. Now, to attain another equilibrium, we can see that decreasing the coefficients $\beta_j$ will restore the equilibrium.$^{[1]}$

Therefore as a general trend, we can say that if we increase the value of $\lambda$ then the magnitude of the coefficients decreases.

Now, if the co-efficients of predictors decrease, then their value in the model decreases. That is, their effect decreases. And thus the flexibility of the model should decrease.

This proof appears appealing, but I have a gut feeling that there are some gaps here and there. If it is correct, good. But if it isn't I would like to know the reasons where this proof fails, and obviously, the correct version of it.

$^{[1]}$: I can attach a plausible explanation on this point, if needed.

In Introduction to Statistical Learning, in the part where ridge regression is explained, the authors say that

As $\lambda$ increases, the flexibility of the ridge regression fit decreases, leading to decreased variance but increased bias.

Here is my take on proving this line:
In ridge regression we have to minimize the sum:$$RSS+\lambda\sum_{j=0}^n\beta_j\\=\sum_{i=1}^n(y_i-\beta_0-\sum_{j=1}^p\beta_jx_{ij})^2+\lambda\sum_{j=1}^p\beta_j^2$$
Here, we can see that a general increase in the $\beta$ vector will decrease $RSS$ and increase the other term. So, in order to minimize the whole term, a kind of equilibrium must be made between the $RSS$ term and the $\lambda\sum_{j=0}^p\beta_j^2$ term. Let their sum be $S$.
Now, if we increase $\lambda$ by $1$, then by using the previous value of the $\beta$ vector, $\lambda\sum_{j=1}^p\beta_j^2$ will increase, whereas $RSS$ will remain the same. Thus $S$ will increase. Now, to attain another equilibrium, we can see that decreasing the coefficients $\beta_j$ will restore the equilibrium.$^{[1]}$

Therefore as a general trend, we can say that if we increase the value of $\lambda$ then the magnitude of the coefficients decreases.

Now, if the co-efficients of predictors decrease, then their value in the model decreases. That is, their effect decreases. And thus the flexibility of the model should decrease.

This proof appears appealing, but I have a gut feeling that there are some gaps here and there. If it is correct, good. But if it isn't I would like to know the reasons where this proof fails, and obviously, the correct version of it.

$^{[1]}$: I can attach a plausible explanation on this point, if needed.

In Introduction to Statistical Learning, in the part where rigde regression is explained, the authors say that

As $\lambda$ increases, the flexibility of the ridge regression fit decreases, leading to decreased variance but increased bias.

Here is my take on proving this line:
In ridge regression we have to minimize the sum:$$RSS+\lambda\sum_{i=0}^n\beta_i\\=\sum_{i=1}^n(y_i-\beta_0-\sum_{j=1}^p\beta_jx_{ij})^2+\lambda\sum_{j=1}^p\beta_j^2$$
Here, we can see that a general increase in the $\beta$ vector will decrease $RSS$ and increase the other term. So, in order to minimize the whole term, a kind of equilibrium must be made between the $RSS$ term and the $\lambda\sum_{i=0}^p\beta_i$ term. Let their sum be $S$.
Now, if we increase $\lambda$ by $1$, then by using the previous value of the $\beta$ vector, $\lambda\sum_{i=1}^p\beta_i$ will increase, whereas $RSS$ will remain the same. Thus $S$ will increase. Now, to attain another equilibrium, we can see that decreasing the coefficients $\beta_j$ will restore the equilibrium.$^{[1]}$

Therefore as a general trend, we can say that if we increase the value of $\lambda$ then the magnitude of the coefficients decreases.

Now, if the co-efficients of predictors decrease, then their value in the model decreases. That is, their effect decreases. And thus the flexibility of the model should decrease.

This proof appears appealing, but I have a gut feeling that there are some gaps here and there. If it is correct, good. But if it isn't I would like to know the reasons where this proof fails, and obviously, the correct version of it.

$^{[1]}$: I can attach a plausible explanation on this point, if needed.

In Introduction to Statistical Learning, in the part where ridge regression is explained, the authors say that

As $\lambda$ increases, the flexibility of the ridge regression fit decreases, leading to decreased variance but increased bias.

Here is my take on proving this line:
In ridge regression we have to minimize the sum:$$RSS+\lambda\sum_{i=0}^n\beta_i\\=\sum_{i=1}^n(y_i-\beta_0-\sum_{j=1}^p\beta_jx_{ij})^2+\lambda\sum_{j=1}^p\beta_j^2$$
Here, we can see that a general increase in the $\beta$ vector will decrease $RSS$ and increase the other term. So, in order to minimize the whole term, a kind of equilibrium must be made between the $RSS$ term and the $\lambda\sum_{i=0}^p\beta_i$ term. Let their sum be $S$.
Now, if we increase $\lambda$ by $1$, then by using the previous value of the $\beta$ vector, $\lambda\sum_{i=1}^p\beta_i$ will increase, whereas $RSS$ will remain the same. Thus $S$ will increase. Now, to attain another equilibrium, we can see that decreasing the coefficients $\beta_j$ will restore the equilibrium.$^{[1]}$

Therefore as a general trend, we can say that if we increase the value of $\lambda$ then the magnitude of the coefficients decreases.

Now, if the co-efficients of predictors decrease, then their value in the model decreases. That is, their effect decreases. And thus the flexibility of the model should decrease.

This proof appears appealing, but I have a gut feeling that there are some gaps here and there. If it is correct, good. But if it isn't I would like to know the reasons where this proof fails, and obviously, the correct version of it.

$^{[1]}$: I can attach a plausible explanation on this point, if needed.

In Introduction to Statistical Learning, in the part where rigde regression is explained, the authors say that

As $\lambda$ increases, the flexibility of the ridge regression fit decreases, leading to decreased variance but increased bias.

Here is my take on proving this line:
In ridge regression we have to minimize the sum:$$RSS+\lambda\sum_{i=0}^n\beta_i\\=\sum_{i=1}^n(y_i-\beta_0-\sum_{j=1}^p\beta_jx_{ij})^2+\lambda\sum_{j=1}^p\beta_j^2$$
Here, we can see that a general increase in the $\beta$ vector will decrease $RSS$ and increase the other term. So, in order to minimize the whole term, a kind of equilibrium must be made between the $RSS$ term and the $\lambda\sum_{i=0}^p\beta_i$ term. Let their sum be $S$.
Now, if we increase $\lambda$ by $1$, then the using the previous value of the $\beta$ vector, $\lambda\sum_{i=1}^p\beta_i$ will increase, whereas $RSS$ will remain the same. Thus $S$ will increase. Now, to attain another equilibrium, we can see that decreasing the coefficients $\beta_j$ will restore the equilibrium.$^{[1]}$

Therefore as a general trend, we can say that if we increase the value of $\lambda$ then the magnitude of the coefficients decreases.

Now, if the co-efficients of predictors decrease, then their value in the model decreases. That is, their effect decreases. And thus the flexibility of the model should decrease.

This proof appears appealing, but I have a gut feeling that there are some gaps here and there. If it is correct, good. But if it isn't I would like to know the reasons where this proof fails, and obviously, the correct version of it.

$^{[1]}$: I can attach a plausible explanation on this point, if needed.

In Introduction to Statistical Learning, in the part where rigde regression is explained, the authors say that

As $\lambda$ increases, the flexibility of the ridge regression fit decreases, leading to decreased variance but increased bias.

Here is my take on proving this line:
In ridge regression we have to minimize the sum:$$RSS+\lambda\sum_{i=0}^n\beta_i\\=\sum_{i=1}^n(y_i-\beta_0-\sum_{j=1}^p\beta_jx_{ij})^2+\lambda\sum_{j=1}^p\beta_j^2$$
Here, we can see that a general increase in the $\beta$ vector will decrease $RSS$ and increase the other term. So, in order to minimize the whole term, a kind of equilibrium must be made between the $RSS$ term and the $\lambda\sum_{i=0}^p\beta_i$ term. Let their sum be $S$.
Now, if we increase $\lambda$ by $1$, then by using the previous value of the $\beta$ vector, $\lambda\sum_{i=1}^p\beta_i$ will increase, whereas $RSS$ will remain the same. Thus $S$ will increase. Now, to attain another equilibrium, we can see that decreasing the coefficients $\beta_j$ will restore the equilibrium.$^{[1]}$

Therefore as a general trend, we can say that if we increase the value of $\lambda$ then the magnitude of the coefficients decreases.

Now, if the co-efficients of predictors decrease, then their value in the model decreases. That is, their effect decreases. And thus the flexibility of the model should decrease.

This proof appears appealing, but I have a gut feeling that there are some gaps here and there. If it is correct, good. But if it isn't I would like to know the reasons where this proof fails, and obviously, the correct version of it.

$^{[1]}$: I can attach a plausible explanation on this point, if needed.