Take the 2-minute tour ×
Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It's 100% free, no registration required.

I have a data set of around 5000 features. For that data I first used Chi Square test for feature selection; after that, I got around 1500 variables which showed significance relationship with the response variable.

Now I need to fit logistic regression on that. I am using glmulti package for R (glmulti package provides efficient subset selection for vlm) but it can use only 30 features at a time, else its performance goes down as the number of rows in my dataset is around 20000.

Is there any other approach or techniques to solve the above problems? If I go by the above method it will take too much time to fit the model.

share|improve this question
If you can fit your dataset in memory of a single machine, I wouldn't call this a "Big Data" problem, specially if you do this in the title of your question –  logc Jun 3 '14 at 14:26

5 Answers 5

up vote 10 down vote accepted

It is not appropriate to do feature screening and then to feed surviving features into a method that does not understand how much data torture was done previously. It is better to use a method that can handle all potential features (e.g., elastic net). Others' suggestions about using data reduction are also excellent ideas.

share|improve this answer
Is there evidence for this? O/w it seems just as much of a heuristic as feature screening. –  Zubin Apr 1 at 23:34
Penalized maximum likelihood estimation takes context into account, e.g., in the lasso a variable's regression coefficient estimate will be smaller if you considered 1000 non-selected variables than if you considered 100 non-selected variables. Otherwise, variables are selected in a biased way and refitting the variable in a second step loses the context. –  Frank Harrell Apr 2 at 11:45

A first approach is to use PCA in order to reduce the dimensionality of the dataset. Try to retain ~97% of the total variance, this may help out quite a bit.

Another option is to use something like stochastic gradient descent, this can be a much faster algorithm and able to fit into R's memory.

EDIT: One problem with R is that you can only use your RAM so if you only have 8 GB of memory then that is what you are limited to. I have run into many problems with this and have since moved onto using python's scikit-learn which seems to handle bigger datasets much better.

A very nice chart which gives some idea of places to start based on your dataset size can be found here: http://3.bp.blogspot.com/-dofu6J0sZ8o/UrctKb69QdI/AAAAAAAADfg/79ewPecn5XU/s1600/scikit-learn-flow-chart.jpg

enter image description here

share|improve this answer
A big concern with using PCA this way is that all the relationship between the response variable and the independent variables could reside in the 3% of total variance that you neglect. There does not seem to be any general way to determine how many principal components to use, either, because the very smallest component could be proportional to the response itself and would thereby constitute the optimal choice of variables to include. –  whuber Jun 3 '14 at 14:17
I think that indeed if you are able to load the dataset in main memory (which I assume is the case considering what you explain), stochastic gradient descent is the first step you should take before trying dimensionality reduction techniques. With Scikit-learn on python (or R, but I'm not a user of this language), this would work just fine. –  Bertrand R Jun 3 '14 at 14:18
I think this is a useful answer, but I think the OP is asking about the logistic regression, and not the feature reduction. Maybe you can address that part of the question in an edition? –  logc Jun 3 '14 at 14:27

As @Frank Harrell already mentioned, using elastic net or LASSO to perform penalized regression with all 5000 features (p) would be a good start for feature selection (one can't simply remove 3500 variables because they are not "statistically significant" with the dependent variable of interest). Either of these methods can be performed using the R package, glmnet.

In order to take into account the relationships shared between the potential predictor variables of interest (p = 5000), I would recommend running a random forest using the randomForest package and/or gradient boosting using the gbm package to assess the relative importance of the potential predictor variables in regards to the binary outcome. With this information, you will be much more prepared to build a more parsimonious logistic regression model.

share|improve this answer
No, it is not correct to do data dredging to decide which parameters to remove from the model. The value of random forests, like elastic net, is that it incorporates the right amount of shrinkage. Starting over with a subset of the variables found in a way that was not masked to $Y$ will cause bias. –  Frank Harrell Jun 9 '14 at 20:49

You can try Vowpal Wabbit: Vowpal Wabbit . It works well with very large datasets and very large number of features.

according to the website:

This is a project started at Yahoo! Research and continuing at Microsoft Research to design a fast, scalable, useful learning algorithm. VW is the essence of speed in machine learning, able to learn from terafeature datasets with ease. Via parallel learning, it can exceed the throughput of any single machine network interface when doing linear learning, a first amongst learning algorithms.

share|improve this answer

We're looking for long answers that provide some explanation and context. Don't just give a one-line answer; explain why your answer is right, ideally with citations. Answers that don't include explanations may be removed.

I assume you are not limited to R, since this is a big data problem you probably shouldn't be. You can try MLlib, which is Apache Spark's scalable machine learning library.

Apache Spark, in turn, is a fast and general engine for in-memory large-scale data processing. These operate on a Hadoop framework that allows for the distributed processing of large data sets across clusters of computers using simple programming models. It is designed to scale up from single servers to thousands of machines, each offering local computation and storage.

Note that 'thousands of machines' is optional(!), you can set it up on your local work/home desktop as well.

Going back to MLlib, it comes with the below algorithms out of the box:

  • K-means clustering with K-means|| initialization.
  • L1- and L2-regularized linear regression.
  • L1- and L2-regularized logistic regression.
  • Alternating least squares collaborative filtering, with explicit ratings or implicit feedback.
  • Naive Bayes multinomial classification.
  • Stochastic gradient descent.

If you are regularly working with big data, you may need to adopt a Hadoop solution.

share|improve this answer

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.