If the dataset is too large to be entirely loaded into memory, how can we do linear regression with the dataset?
If data is large, iterative method is better than direct method to solve the linear system.
Details can be found in this post
In addition, stochastic gradient decent can be used to learn from the very large data set. I also discussed it on my answer linked above. The idea is to approximate the gradient from a subset of the data. Which can be implemented in parallel.
If your data is too tall, then a standard technique is batching, where you update the loss function for say, 1000 points at a time. This is how stochastic gradient descent works.
If your data is also too wide, then I would think a similar kind of batching procedure would work, where you also select a subset of features to update at any given time. This would be analogous to how dropout works in neural networks.