If the dataset is too large to be entirely loaded into memory, how can we do linear regression with the dataset?
3
-
3$\begingroup$ A sequential method for estimating a regression model is to "build up" an array of product-moments by sequentially reading in segments of a file. Store the cumulative results for $\sum X$, $\sum Y $, $\sum XY$, $\sum X^2$ and $\sum Y^2$. $\hat{\beta}_1 = \sum XY / \sum XX$ and the $\hat{\beta}_0 = \bar{Y} - \bar{X}\hat{\beta}_1$. Multivariate analogues are easily extended from this. $\endgroup$– AdamOFeb 12, 2018 at 18:09
-
2$\begingroup$ This is in fact the method that is implemented in SAS PROC GLM, among others. It makes use of the fact that $X^TX$ = \sum_i x_ix_i^T$ among other things; you can construct the r.h.s. one row at a time. $\endgroup$– jbowmanFeb 12, 2018 at 19:08
-
$\begingroup$ Odd.. Three sensible answer and not a single upvote... Fixed. $\endgroup$– usεr11852Feb 18, 2018 at 21:35
Add a comment
|
3 Answers
$\begingroup$
$\endgroup$
1
For larger dataset, we use stochastic gradient descent or batch-gradient descent. But using these may give a optimum value that is close enough. I would suggest you to use batch-gradient descent as it gives better optimum values rather than stochastic gradient descent.
-
$\begingroup$ And I forgot to mention. When using stochastic gradient descent, we need to randomly use the data. If the data is some particular order or sequence, then it may cause some bias issues. $\endgroup$ Feb 14, 2018 at 11:23
$\begingroup$
$\endgroup$
1
If data is large, iterative method is better than direct method to solve the linear system.
Details can be found in this post
https://stats.stackexchange.com/a/278779/113777
In addition, stochastic gradient decent can be used to learn from the very large data set. I also discussed it on my answer linked above. The idea is to approximate the gradient from a subset of the data. Which can be implemented in parallel.
answered Feb 12, 2018 at 17:54
-
$\begingroup$ I think you mean sequential rather than cumulative. Also if a link-out answer is adequate maybe it can be closed as off-topic? But I do not think this answers the question. $\endgroup$– AdamOFeb 12, 2018 at 18:10
$\begingroup$
$\endgroup$
If your data is too tall, then a standard technique is batching, where you update the loss function for say, 1000 points at a time. This is how stochastic gradient descent works.
If your data is also too wide, then I would think a similar kind of batching procedure would work, where you also select a subset of features to update at any given time. This would be analogous to how dropout works in neural networks.
