Timeline for Does ridge regression with a duplicate column actually halve the coefficient?
Current License: CC BY-SA 4.0
9 events
| when toggle format | what | by | license | comment | |
|---|---|---|---|---|---|
| Oct 9 at 16:40 | comment | added | Steven Gubkin | If you permit me to change $\lambda$, then I can achieve any model parameter in between the psuedoinverse (OLS) parameter value and $0$, so the statement wouldn't be worth very much... | |
| Oct 9 at 16:33 | comment | added | John Radinger | I agree that a counterexample can be produced (it is easily doable) via simulations if we choose the same $\lambda$ for both ridge regressions. But an alternative (and reasonable) interpretation is that the two coefficients are exactly one half of a coefficient for some other choice of $\lambda$ (tougher to show a counterexample for). | |
| Oct 9 at 16:23 | comment | added | John Radinger | Sure, but then, why does line 3 follow? The first two models include a perturbed covariate $X_3$, but the model with a single coefficient must not --- I'd think it'd be helpful if you explicitly it wrote which covariates are the same and which are different in the 3 ridge regression models you presented. | |
| Oct 9 at 16:22 | comment | added | Steven Gubkin | There isn't anything wrong with my argument. I will code up an explicit counterexample and post the results momentarily in a new answer. | |
| Oct 9 at 16:20 | comment | added | John Radinger | Note that the above excerpt I cited is not from ESL, but rather from a paper published in Technometrics (ncbi.nlm.nih.gov/pmc/articles/PMC9410599), so it would be surprising if they're expressing some intuition (which they would explicitly state, I'd imagine) rather than state something which is mathematically clear but wrong. | |
| Oct 9 at 16:20 | comment | added | Steven Gubkin | This circumlocution with $\epsilon$ is not really needed: my original argument is fine. I just hope that it helps you to see that the argument is valid. | |
| Oct 9 at 16:19 | comment | added | Steven Gubkin | @JohnRadinger If it helps imagine that $X_1$ and $X_2$ are identical and $X_3$ is just ever so slightly perturbed (you could, for example, add $\epsilon$ to only the first coordinate). The theorem says that since $X_1$ and $X_2$ are identical their regression coefficients should be equal to half of what they are if only one were included in the model. Line 1 of my three equalities follows. These ridge regression coefficients should be continuous in $\epsilon$, so as $\epsilon \to 0$ we have the three equalities I started with. | |
| Oct 9 at 14:32 | comment | added | John Radinger | Sorry, I don't think your example is correct. If all 3 covariates are identical, then we shouldn't assume that $\beta_1 + \beta_2 = \beta_{12}$ because $\beta'_3$ should also be identical to $\beta_{12}$ and we should be distributing $\beta_1 + \beta_2 + \beta_3$ equally over $\beta_{12}$ and $\beta_3'$. The natural generalization of the theorem I'm discussing is to cluster all columns into identical groups, and then if the columns of each group was replaced with just a single unique column, that new ridge coefficient should be the sum of the (identical) original ridge coefs in each group. | |
| Oct 9 at 11:40 | history | answered | Steven Gubkin | CC BY-SA 4.0 |