Faster option than glmnet for elastic net regularized regression [closed]

Question

I attempted to predict crime category from its X and Y coordinates based on San Francisco Crime Statistics (https://www.kaggle.com/c/sf-crime). And it turned out that glmnet is pretty slow for fitting on this dataset: when I fit on a sample of 100k observations, it takes about 3 minutes. If I extrapolate that time to the whole dataset (878k observations) - the whole fit will take about 25 minutes.

So, my questions are

1. What is the maximum data set size to work in R interactively?
2. How faster would be other languages (Python, Java) on the similar task?
3. Does such performance mean that I should not at all attempt at all to solve such a big problem in R and choose instead other language?

Answer 1

First, you have a good idea that you can get a feel for things with a smaller sample, to work out the kinks in your approach. And it's definitely true, as you've noticed, that things that take a while to run really break up your focus. It's annoying. But...

Second, you have to define "interactively". Do you mean instantaneously, 1 second, 10 seconds, or what?

Third, you need to account for your hardware, the software you're using, how much data you have, and what algorithm you're executing.

For example, doing an lm on 100K rows of data might be "interactive" for you. Obviously glmnet is doing a lot more than that.

In terms of other languages, Python or Java may be faster if you have a lot of code outside of glmnet that you're executing. As one of the comments says, glmnet is coded in C or Fortran and will be as fast as possible. If you're doing looping or a lot of code around your glmnet, it's easier in R to do something inefficient than it is in programming languages not designed for statistics like Python or Java.

It's possible to parallelize some algorithms. I'm not sure if glmnet is one of them. If you find a language that has a parallelized version -- say one that uses multiple cores, and you're running on a machine with multiple cores and enough RAM -- that will speed things up sub-linearly. Four cores won't be four times as fast, but it should be 2-3x faster.

So, the answer is "no, probably not". Your algorithm will usually trump anything else, and perhaps glmnet alone isn't the best option.

Answer 2

Following @Wayne's comments on parallelization, see Zhou et al. (2014), "A Reduction of the Elastic Net to Support Vector Machines with an Application to GPU Computing", arXiv:1409.1976. They say

The state-of-the-art single-core implementation for solving the Elastic Net problem is the glmnet package developed by Friedman. Mostly written in Fortran language, glmnet adopts the coordinate gradient descent strategy and is highly optimized. As far as we know, it is the fastest off-the-shelf solver for the Elastic Net. Due to its inherent sequential nature, the coordinate descent algorithm is extremely hard to parallelize.

but come up with a different approach, showing the elastic net is equivalent a type of support vector machine (which can be parallelized):

[...] we take inspiration from recent work on machine learning reductions [15, 19] and we reduce the Elastic Net to the squared hinge-loss SVM (without a bias term). We show that this reduction is exact and extremely efficient in practice. The resulting algorithm, which we refer to as Support Vector Elastic Net (SVEN), naturally takes advantage of the vast existing work on parallel SVMs, immediately providing highly efficient Elastic Net and Lasso implementations on GPUs, multi-core CPUs and distributed systems

Note that hyper-parameter selection by cross-validation can of course be parallelized (and is implemented for the shrinkage hyper-parameter in cv.glmnet—see executing glmnet in parallel in R).

The Matlab code to fit a Support Vector Elastic Net is available at https://bitbucket.org/mlcircus/sven/. You could give that a try!