Timeline for Linear regression on a large dataset
Current License: CC BY-SA 4.0
9 events
| when toggle format | what | by | license | comment | |
|---|---|---|---|---|---|
| Aug 24, 2023 at 13:05 | comment | added | hobbs | Too large to be loaded into memory, needs to be processed streaming: fine. But that doesn't mean you can only read it once. You can make one pass computing summary statistics like min, max, and mean, and a second pass where you have those numbers available. | |
| Aug 24, 2023 at 11:16 | comment | added | quarague | What software are you using? Some programs are a lot better at handling large data sets than others or have suitable functions to do it bit by bit. | |
| Aug 24, 2023 at 9:50 | comment | added | Christian Hennig | Why do you need to scale the data? Linear regression is affine equivariant, so you'll get equivalent results whether you scale or not. | |
| Aug 24, 2023 at 7:23 | history | became hot network question | |||
| Aug 24, 2023 at 6:27 | history | edited | Stephan Kolassa |
edited tags
|
|
| Aug 24, 2023 at 6:27 | answer | added | Stephan Kolassa | timeline score: 7 | |
| Aug 24, 2023 at 0:55 | comment | added | Galen | If you've got that much data do you really need to use all of it if you're just doing simple linear regression? Try some simple random samples of $10^1$,$10^2$, $10^3$, $10^4$ to see if increasing the sample size really matters. There is diminishing returns to the improvement of the standard errors of your parameters. | |
| Aug 23, 2023 at 23:50 | history | edited | z611 | CC BY-SA 4.0 |
deleted 52 characters in body
|
| Aug 23, 2023 at 23:05 | history | asked | z611 | CC BY-SA 4.0 |