Linear regression for large dataset If the dataset is too large to be entirely loaded into memory, how can we do linear regression with the dataset? 
 A: If data is large, iterative method is better than direct method to solve the linear system.
Details can be found in this post
https://stats.stackexchange.com/a/278779/113777
In addition, stochastic gradient decent can be used to learn from the very large data set. I also discussed it on my answer linked above. The idea is to approximate the gradient from a subset of the data. Which can be implemented in parallel.
A: If your data is too tall, then a standard technique is batching, where you update the loss function for say, 1000 points at a time. This is how stochastic gradient descent works. 
If your data is also too wide, then I would think a similar kind of batching procedure would work, where you also select a subset of features to update at any given time. This would be analogous to how dropout works in neural networks. 
A: For larger dataset, we use stochastic gradient descent or batch-gradient descent.
But using these may give a optimum value that is close enough. I would suggest you to use batch-gradient descent as it gives better optimum values rather than stochastic gradient descent.
