Extremely fast regularized linear model libraries I have a huge dataset (millions of rows, thousands of columns) and glmnet Lasso regularised regression is too slow.
I am wondering what other libraries there are that try to implement regularised linear model estimation extremely efficiently? I don't necessarily need a distributed solution (it can estimate on one CPU), and it is fine for me to load all data to RAM (so an R solution may be fine, if one exists that's faster somehow than glmnet). It also does not necessarily need to be Lasso or Elastic Net.
I can use java, python or R.
My data is pretty messy, some columns are sparse, some have heavy autocorrelation, etc. 
 A: Some form gradient descent algorithm will be fastest on your problem.
There are a number of variants of gradient descent algorithms and and a number of different algorithms.
The most basic variant is Stochastic Gradient Descent (SGD).
Most of the major machine learning libraries/systems will have a version of this.
For massive datasets that do not fit into memory Vovpal Wabbit is probably the go,
but for your data sets most of the major implementations should be fine.
Enhancements to SGD include Pegasos, PassiveAggressive algorithms, Confidence Wighted Learning, Adagrad, AROW, and Natural Gradient Descent.
If you use Python then the scikit library has SGD and Passive Agrresive Preceptrons. I have used them very succesfully for datasets like you describe.
Training should take only seconds or possibly minutes depending on the particulars of your data and the settings you use.
If your inputs are sparse, make sure the algorithm your algorithm is using sparse data structures to represent them. This could be what made glmnet slow.
