Timeline for The limit of "unit-variance" ridge regression estimator when $\lambda\to\infty$
Current License: CC BY-SA 3.0
33 events
| when toggle format | what | by | license | comment | |
|---|---|---|---|---|---|
| Apr 22, 2018 at 19:46 | history | edited | amoeba | CC BY-SA 3.0 |
added 84 characters in body
|
| Apr 19, 2018 at 21:51 | comment | added | Sextus Empiricus | That looks cool. I will have to let this rest for a while and have to try to 'get it' from more different angles than just geometrical-algebraic, and try to see practical meaning and a larger picture. | |
| Apr 19, 2018 at 21:24 | comment | added | amoeba | @MartijnWeterings See my answer to stats.stackexchange.com/questions/331264. | |
| Apr 19, 2018 at 17:44 | comment | added | amoeba | @MartijnWeterings Actually one can continue the ridge path in the other direction (beyond OLS) too. There the RSS ellipses will be touching the circles from the inside. This should correspond to the negative lambdas between 0 and $-s^2_\mathrm{min}$. | |
| Apr 19, 2018 at 15:30 | comment | added | amoeba | @MartijnWeterings I see why you suggest that but I did not want to extend PLS vector in the opposite direction because PLS has a "natural" direction (the sign of $X^\top y$), whereas PCA arguably does not... I will think about it. | |
| Apr 19, 2018 at 15:24 | comment | added | Sextus Empiricus | In the new figure I would have added also the PLS vector and then placed the label 'neg ridge path' on the other side. | |
| Apr 19, 2018 at 15:18 | comment | added | amoeba | @MartijnWeterings YES!!! I did not realize this, but of course it's like that. Check out the new figure. I decided to add some colors because it's becoming very busy. | |
| Apr 19, 2018 at 15:17 | history | edited | amoeba | CC BY-SA 3.0 |
added 117 characters in body
|
| Apr 19, 2018 at 15:07 | comment | added | Sextus Empiricus | It would add to the visualization. I imagine the current three RR path points (where the circle and ellipsoid touch) continuing downwards to the right and eventually, at infinity, the circle $|\beta|^2=t_{\infty}$ and ellipsoid $|X (\beta - \hat\beta)|^2 =RSS$ should 'touch' in direction of the the spot where the circle $|\beta|^2=t_{pca}$ touches the ellipsoid $|X \beta|^2 =1$ | |
| Apr 19, 2018 at 14:58 | comment | added | amoeba | @MartijnWeterings Yes -- and wow, path going to the lower right is a great visualization idea!! I've been imagining it going into the sector between PLS and PCA lines on my figure and that part of the figure is very crowded (and the path would need to "bounce" back at zero, which does not look so good). But going downwards solves all these problems! | |
| Apr 19, 2018 at 14:55 | comment | added | Sextus Empiricus | ah, ok, then $\lambda^*$ and $1+\mu^*$ go to $\pm$ infinity but their ratio remains $s_{max}^2$. In any case, the negative ridge regression path should be in the (negative) sector between the PLS and PCA vectors such that their projection onto the ellipse $|X\beta=1|$ is between the points PLS and PCA. (the norm going to infinity makes sense as the $\mu$ goes to infinity as well, so the path continues to the lower right, initially tangent to, negative, PLS and eventually to PCA) | |
| Apr 19, 2018 at 14:18 | comment | added | amoeba | @MartijnWeterings No, it goes to $-s^2_\mathrm{max}$ which is the squared largest singular value of X. Then $X^\top X+\lambda I$ has one eigenvalue going to zero, meaning that the inverse has one eigenvalue growing to infinity, and that's the eigenvalue corresponding to PC1. That's why you get PC1 in the limit. | |
| Apr 19, 2018 at 14:06 | comment | added | Sextus Empiricus | Near the PCA point both the $\lambda^*$ and $1+\mu^*$ (for the normalized RR) grow to $\pm$ infinity. Possibly $\lambda = \frac{\lambda^*}{1+\mu^*}$ goes to zero. | |
| Apr 19, 2018 at 13:34 | comment | added | amoeba | @MartijnWeterings That's a good suggestion, but I am not quite sure how the negative RR path will look like :-) I mean the non-normalized RR with negative $\lambda$. I think it should "start" at zero for $\lambda=-\infty$, go initially in the PLS direction and then approach PCA direction but with norm growing to infinity for $\lambda$ approaching $-\lambda_\mathrm{max}$... | |
| Apr 19, 2018 at 13:31 | comment | added | Sextus Empiricus | I liked the graph a lot, maybe because it was better than mine (and I like grayscale). Changes might be detrimental, although a change of the label is just minor, yet it is quite clear already (in my view). I believe that when something is original and good then it is best (so much editing happens nowadays, such that raw ideas barely reach the suface, even in a simple q&a site). When we are "criticizing" anyway: one thing that I imagined is that you could add the negative RR path as well in this pretty plot (but don't listen to me as the plot is fine already). | |
| Apr 19, 2018 at 12:59 | history | edited | amoeba | CC BY-SA 3.0 |
added 22 characters in body
|
| Apr 19, 2018 at 12:55 | comment | added | amoeba | @Ryan Maybe I just need to write a figure caption :) How would you rename "RR path"? Would "Ridge path" be clearer? BTW, it's $\hat{\boldsymbol\beta}_\lambda$ without star in my notation; star was supposed to mean that it's coming from the constrained problem. Also BTW, if you like to post something based on your review of continuum regression, I'd encourage you to post another answer. I find it interesting how the solutions to "my" problem live on the $|X\beta|=1$ ellipse, but the solutions to the continuum regression problem live on the $|\beta|=1$ circle (with one-to-one correspondence). | |
| Apr 19, 2018 at 12:40 | comment | added | Ryan Simmons | I spent some time looking for papers about continuum regression with the extra constraint concerning the XB projection, but as of yet have found nothing (but this field does tend to use an idiosyncratic notation at times so perhaps I've overlooked something). In the meantime, I really like your new figure, but perhaps it might be best to more clearly indicate what "RR path" is? It may not be clear to all readers that this corresponds to $\hat{\boldsymbol\beta}_\lambda^*$ as $\lambda$ changes from 0 to infinity. | |
| Apr 17, 2018 at 20:40 | history | edited | amoeba | CC BY-SA 3.0 |
added 53 characters in body
|
| Apr 17, 2018 at 15:09 | history | edited | amoeba | CC BY-SA 3.0 |
added 157 characters in body
|
| Apr 17, 2018 at 11:36 | comment | added | amoeba |
@Ryan I searched for "continuum regression" on our site and found nothing at all (well, now this answer comes up in the search). This is my measure of how "obscure" a concept is :-) Even though Stone & Brooks 1990 has 500+ citations on google scholar which is actually a lot.
|
|
| Apr 17, 2018 at 11:27 | history | edited | amoeba | CC BY-SA 3.0 |
removed nu
|
| Apr 17, 2018 at 10:41 | history | edited | amoeba | CC BY-SA 3.0 |
added 167 characters in body
|
| Apr 17, 2018 at 7:52 | history | edited | amoeba | CC BY-SA 3.0 |
deleted 10 characters in body
|
| Apr 16, 2018 at 22:22 | comment | added | Ryan Simmons | I do not have a chemometrics background; I'm a fairly run-of-the-mill biostatistician. But penalized regression in general has become a bit of a research interest for me, and "continuum regression" (agree with you on the name!) and some of the related ideas that come up in chemometrics I rather stumbled across by chance. I am still new to this area of study, but have a pile of about a dozen papers I am planning on working my way through. I will certainly respond to this question if I find something of more specific relevance! | |
| Apr 16, 2018 at 20:43 | history | edited | amoeba | CC BY-SA 3.0 |
added 203 characters in body
|
| Apr 16, 2018 at 20:35 | history | edited | amoeba | CC BY-SA 3.0 |
added 203 characters in body
|
| Apr 16, 2018 at 19:36 | comment | added | amoeba | By the way, do you have a chemometrics background? To be honest, most of this field seems to have been developed in isolation from the "mainstream" statistics. | |
| Apr 16, 2018 at 19:36 | comment | added | amoeba | @Ryan, thanks. I only learned about "continuum regression" because I accidentally saw your comment under Benoit's question. De Jong et al. 2001 is among the first results in the Google Scholar search I linked to; I looked at several papers there but haven't seen this formulation with ridge loss constrained to have a projection with unit variance anywhere. Do let me know if you saw it somewhere! For me (with more of a machine learning background) this is a much more natural way to introduce this framework than what I see in the "continuum regression" literature (I also dislike the name btw). | |
| Apr 16, 2018 at 15:02 | comment | added | Ryan Simmons | In case that link dies, the full citation is: Sijmen de Jong, Barry M. Wise, N. Lawrence Ricker. "Canonical partial least squares and continuum power regression." Journal of Chemometrics, 2001; 15: 85-100. doi.org/10.1002/… (3/3) | |
| Apr 16, 2018 at 14:50 | comment | added | Ryan Simmons | One paper you may want to read is de Jong et al. (2001). Their formulation of "canonical PLS" seems on a quick glance to be equivalent to yours, though I admit I haven't rigorously compared the math yet (they also provide a review of several other PLS-PCA generalizations in the same vein). But it may be insightful to see how they have explicated the problem. (2/3) | |
| Apr 16, 2018 at 14:46 | comment | added | Ryan Simmons | "Continuum regression" seems to be one of a surprisingly broad category of techniques aimed at unifying PLS and PCA within a common framework. I had never heard about it, incidentally, until researching negative ridge (I provide a link to the Bjorkstron & Sundberg, 1999, paper in the first comment of the negative ridge question you link to), though it seems to be rather widely discussed in the chemometric literature. There must be some historical reason why it has developed seemingly in isolation from other fields of statistics. (1/3) | |
| Apr 16, 2018 at 14:07 | history | answered | amoeba | CC BY-SA 3.0 |