Timeline for Python package for making a rank-deficient sparse matrix full rank
Current License: CC BY-SA 4.0
7 events
| when toggle format | what | by | license | comment | |
|---|---|---|---|---|---|
| May 20, 2023 at 22:35 | comment | added | whuber♦ | I am not judging your code, which I did not run, but reacting to your description, to wit: "Remove any column whose (absolute) max correlation is greater than the threshold." That implies you will drop all columns when all correlations exceed the threshold. If that's not what you are proposing, then please edit your post to clarify what you do suggest the OP do. However, as a general principle in regression, dropping columns based solely on pairwise correlations is a notably bad idea, so the details don't really matter. | |
| May 20, 2023 at 20:20 | comment | added | tkunk | @whuber You also wrote that it's possible to remove every column, when it's not, so it's likely you don't understand enough about it to judge it. | |
| May 20, 2023 at 18:55 | comment | added | emonigma | Thank you. In fact, this is the procedure I'm already using, and I am looking for a better one that keeps some of those variables. I updated the question with an explanation. | |
| May 20, 2023 at 15:27 | comment | added | whuber♦ | As I wrote, that's a really poor procedure and it doesn't meet the requirements of the question. | |
| May 20, 2023 at 15:22 | comment | added | tkunk | @whuber It removes correlated variables, which is exactly what I said it does. If N features are correlated with each other, it'll remove N - 1 of them. | |
| May 19, 2023 at 19:16 | comment | added | whuber♦ | That sounds like an extremely poor statistical procedure! Moreover, it won't necessarily solve the problem: how do you maintain the rank of the matrix? (It's possible your procedure would remove every column!) Wouldn't the OP be better off with a recommendation to do something that worked better? | |
| May 19, 2023 at 18:21 | history | answered | tkunk | CC BY-SA 4.0 |