Timeline for Python package for making a rank-deficient sparse matrix full rank
Current License: CC BY-SA 4.0
23 events
| when toggle format | what | by | license | comment | |
|---|---|---|---|---|---|
| May 25, 2023 at 8:19 | comment | added | emonigma | @seanv507 Yes, it was an XY problem, and running a regularised regression solved it. I added it as a comment to the accepted answer. Thanks! | |
| May 25, 2023 at 8:15 | vote | accept | emonigma | ||
| May 22, 2023 at 17:42 | comment | added | seanv507 | I think it still sounds like an XY problem. If your goal is to perform a regression with a rank deficient matrix you are better off using a regularised regression algorithm such as ridge regression (L2) or lasso (L1) coefficient regularisation. L2 is what I would naturally do for correlated - eg if NY city and NYC are 100% correlated then you will have 1/2 weight on each. | |
| May 22, 2023 at 13:04 | answer | added | Sycorax♦ | timeline score: 6 | |
| May 22, 2023 at 12:59 | history | reopened |
Adrian Keister utobi Sycorax♦ |
||
| May 22, 2023 at 12:48 | history | left closed in review |
User1865345 whuber♦ |
Original close reason(s) were not resolved | |
| May 22, 2023 at 8:02 | history | edited | emonigma | CC BY-SA 4.0 |
added 148 characters in body
|
| May 22, 2023 at 7:07 | comment | added | emonigma | @Sycorax Fair point. I've edited the question to remove mention of dropping columns, and I believe addresses the XY problem. What I want is to invert the matrix keeping the highest possible number of dimensions. | |
| S May 22, 2023 at 7:06 | review | Reopen votes | |||
| May 22, 2023 at 12:48 | |||||
| S May 22, 2023 at 7:06 | history | edited | emonigma | CC BY-SA 4.0 |
removed mention of dropping columns
Added to review
|
| May 21, 2023 at 16:43 | comment | added | Sycorax♦ | Let me be more clear. Removing features with high correlation is, at best, only a coincidental solution to producing a full-rank matrix. At worst, it arbitrarily removes features without regard to anything you might want to learn from the model, with the kicker that it also hasn’t solved the problem of producing a full-rank matrix! In other words, this is an XY Problem. Rank-revealing factorizations like SVD or QR are examples of the tools for finding a full-rank matrix. | |
| May 20, 2023 at 22:34 | history | left closed in review | whuber♦ | Original close reason(s) were not resolved | |
| S May 20, 2023 at 19:00 | review | Reopen votes | |||
| May 20, 2023 at 22:34 | |||||
| S May 20, 2023 at 19:00 | history | edited | emonigma | CC BY-SA 4.0 |
added explanation
Added to review
|
| May 20, 2023 at 18:54 | comment | added | emonigma | Thank you @Sycorax, I have updated the question with an explanation. | |
| May 20, 2023 at 18:53 | history | edited | emonigma | CC BY-SA 4.0 |
added explanation
|
| May 19, 2023 at 18:53 | history | closed | Sycorax♦ | Needs details or clarity | |
| May 19, 2023 at 18:53 | history | notice removed | Sycorax♦ | ||
| May 19, 2023 at 18:53 | comment | added | Sycorax♦ | Removing pairwise correlated columns is not the same as finding a full-rank matrix, because three or more columns can have low correlation but together be collinear. (Consider the dummy variable trap.) Which goal are you trying to achieve, retaining columns with low correlation or dropping columns until your matrix is a full-rank matrix? Or something else? | |
| May 19, 2023 at 18:21 | answer | added | tkunk | timeline score: -3 | |
| May 16, 2023 at 8:43 | history | notice added | emonigma | Draw attention | |
| May 16, 2023 at 8:43 | history | edited | emonigma | CC BY-SA 4.0 |
added 29 characters in body
|
| May 4, 2023 at 19:34 | history | asked | emonigma | CC BY-SA 4.0 |