5
$\begingroup$

I want to run linear regression on a dataset who is too large to be loaded into memory. I intend to simply calculate

$$\left(\sum_{i=1}^n x_ix_i^T\right)^{-1} \cdot \left(\sum_{i=1}^n x_iy_i\right)$$ but one issue is that my input data is all different scales. If the scale is sufficiently small, it's OK as I can simply scale the dataset to lie in $[-1, 1]$ all at once, using.

X = 2 * (X-X.min())/(X.max()-X.min()) - 1

But how can I scale it if I'm doing it iteratively? We won't know the max and min apriori.

Improve this question
$\endgroup$
4
  • 11
    $\begingroup$ If you've got that much data do you really need to use all of it if you're just doing simple linear regression? Try some simple random samples of $10^1$,$10^2$, $10^3$, $10^4$ to see if increasing the sample size really matters. There is diminishing returns to the improvement of the standard errors of your parameters. $\endgroup$
    – Galen
    Commented Aug 24, 2023 at 0:55
  • 1
    $\begingroup$ Why do you need to scale the data? Linear regression is affine equivariant, so you'll get equivalent results whether you scale or not. $\endgroup$
    – Christian Hennig
    Commented Aug 24, 2023 at 9:50
  • $\begingroup$ What software are you using? Some programs are a lot better at handling large data sets than others or have suitable functions to do it bit by bit. $\endgroup$
    – quarague
    Commented Aug 24, 2023 at 11:16
  • 3
    $\begingroup$ Too large to be loaded into memory, needs to be processed streaming: fine. But that doesn't mean you can only read it once. You can make one pass computing summary statistics like min, max, and mean, and a second pass where you have those numbers available. $\endgroup$
    – hobbs
    Commented Aug 24, 2023 at 13:05

1 Answer 1

Reset to default
7
$\begingroup$

There are specialized algorithms that allow estimating a least squares problem iteratively, so you don't have to have the entire dataset in memory, but can update your solution by iterating over the data. Take a look at LSQR or LSMR, depending on the properties of your design matrix $X$.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.