Logistic Regression on Big Data I have a data set of around 5000 features. For that data I first used Chi Square test for feature selection; after that, I got around 1500 variables which showed significance relationship with the response variable. 
Now I need to fit logistic regression on that. I am using glmulti package for R (glmulti package provides efficient subset selection for vlm) but it can use only 30 features at a time, else its performance goes down as the number of rows in my dataset is around 20000.
Is there any other approach or techniques to solve the above problems? If I go by the above method it will take too much time to fit the model.
 A: As @Frank Harrell already mentioned, using elastic net or LASSO to perform penalized regression with all 5000 features (p) would be a good start for feature selection (one can't simply remove 3500 variables because they are not "statistically significant" with the dependent variable of interest). Either of these methods can be performed using the R package, glmnet.
In order to take into account the relationships shared between the potential predictor variables of interest (p = 5000), I would recommend running a random forest using the randomForest package and/or gradient boosting using the gbm package to assess the relative importance of the potential predictor variables in regards to the binary outcome. With this information, you will be much more prepared to build a more parsimonious logistic regression model.
A: It is not appropriate to do feature screening and then to feed surviving features into a method that does not understand how much data torture was done previously.  It is better to use a method that can handle all potential features (e.g., elastic net).  Others' suggestions about using data reduction are also excellent ideas.
A: A first approach is to use PCA in order to reduce the dimensionality of the dataset. Try to retain ~97% of the total variance, this may help out quite a bit. 
Another option is to use something like stochastic gradient descent, this can be a much faster algorithm and able to fit into R's memory. 
EDIT: One problem with R is that you can only use your RAM so if you only have 8 GB of memory then that is what you are limited to. I have run into many problems with this and have since moved onto using python's scikit-learn which seems to handle bigger datasets much better. 
A very nice chart which gives some idea of places to start based on your dataset size can be found here: http://3.bp.blogspot.com/-dofu6J0sZ8o/UrctKb69QdI/AAAAAAAADfg/79ewPecn5XU/s1600/scikit-learn-flow-chart.jpg

A: I assume you are not limited to R, since this is a big data problem you probably shouldn't be. You can try MLlib, which is Apache Spark's scalable machine learning library.
Apache Spark, in turn, is a fast and general engine for in-memory large-scale data processing. These operate on a Hadoop framework that allows for the distributed processing of large data sets across clusters of computers using simple programming models. It is designed to scale up from single servers to thousands of machines, each offering local computation and storage.
Note that 'thousands of machines' is optional(!), you can set it up on your local work/home desktop as well.
Going back to MLlib, it comes with the below algorithms out of the box:

*

*K-means clustering with K-means|| initialization.

*L1- and L2-regularized linear regression.

*L1- and L2-regularized logistic regression.

*Alternating least squares collaborative filtering, with explicit ratings or implicit feedback.

*Naive Bayes multinomial classification.

*Stochastic gradient descent.

If you are regularly working with big data, you may need to adopt a Hadoop solution.
A: You can try Vowpal Wabbit: Vowpal Wabbit
. It works well with very large datasets and very large number of features. 
according to the website:

This is a project started at Yahoo! Research and continuing at
  Microsoft Research to design a fast, scalable, useful learning
  algorithm. VW is the essence of speed in machine learning, able to
  learn from terafeature datasets with ease. Via parallel learning, it
  can exceed the throughput of any single machine network interface when
  doing linear learning, a first amongst learning algorithms.

