9
$\begingroup$

I have a data set of around 5000 features. For that data I first used Chi Square test for feature selection; after that, I got around 1500 variables which showed significance relationship with the response variable.

Now I need to fit logistic regression on that. I am using glmulti package for R (glmulti package provides efficient subset selection for vlm) but it can use only 30 features at a time, else its performance goes down as the number of rows in my dataset is around 20000.

Is there any other approach or techniques to solve the above problems? If I go by the above method it will take too much time to fit the model.

Improve this question
$\endgroup$
2
  • 8
    $\begingroup$ If you can fit your dataset in memory of a single machine, I wouldn't call this a "Big Data" problem, specially if you do this in the title of your question $\endgroup$
    – logc
    Jun 3, 2014 at 14:26
  • $\begingroup$ I'm using sklearn's LogisticRegression and it solves a 4000 features, 20,000 rows problem in about a minute on my laptop. $\endgroup$
    – Thomas Ahle
    Jul 25, 2019 at 19:38

5 Answers 5

Reset to default
13
$\begingroup$

It is not appropriate to do feature screening and then to feed surviving features into a method that does not understand how much data torture was done previously. It is better to use a method that can handle all potential features (e.g., elastic net). Others' suggestions about using data reduction are also excellent ideas.

Improve this answer
$\endgroup$
4
  • 1
    $\begingroup$ Is there evidence for this? O/w it seems just as much of a heuristic as feature screening. $\endgroup$
    – Zubin
    Apr 1, 2015 at 23:34
  • 2
    $\begingroup$ Penalized maximum likelihood estimation takes context into account, e.g., in the lasso a variable's regression coefficient estimate will be smaller if you considered 1000 non-selected variables than if you considered 100 non-selected variables. Otherwise, variables are selected in a biased way and refitting the variable in a second step loses the context. $\endgroup$
    – Frank Harrell
    Apr 2, 2015 at 11:45
  • 1
    $\begingroup$ Could you elaborate on what you mean by biased here? I am wondering, because in a trivial sense the lasso is always biased, so you have to refer to an additional bias. Also, there are some two-stage approaches with reasonable properties, e.g. pdfs.semanticscholar.org/d90a/… $\endgroup$
    – jmb
    Oct 28, 2016 at 8:57
  • 2
    $\begingroup$ The lasso purposely biases coefficients towards zero to prevent overfitting. Unpenalized parameters lead to predicted values that are too extreme. Fitting "selected" variables in an unpenalized model will undo the needed lasso bias, creating much overfitting of predicted values. $\endgroup$
    – Frank Harrell
    Oct 28, 2016 at 11:54
10
$\begingroup$

A first approach is to use PCA in order to reduce the dimensionality of the dataset. Try to retain ~97% of the total variance, this may help out quite a bit.

Another option is to use something like stochastic gradient descent, this can be a much faster algorithm and able to fit into R's memory.

EDIT: One problem with R is that you can only use your RAM so if you only have 8 GB of memory then that is what you are limited to. I have run into many problems with this and have since moved onto using python's scikit-learn which seems to handle bigger datasets much better.

A very nice chart which gives some idea of places to start based on your dataset size can be found here: http://3.bp.blogspot.com/-dofu6J0sZ8o/UrctKb69QdI/AAAAAAAADfg/79ewPecn5XU/s1600/scikit-learn-flow-chart.jpg

enter image description here

Improve this answer
$\endgroup$
4
  • 8
    $\begingroup$ A big concern with using PCA this way is that all the relationship between the response variable and the independent variables could reside in the 3% of total variance that you neglect. There does not seem to be any general way to determine how many principal components to use, either, because the very smallest component could be proportional to the response itself and would thereby constitute the optimal choice of variables to include. $\endgroup$
    – whuber ♦
    Jun 3, 2014 at 14:17
  • 1
    $\begingroup$ I think that indeed if you are able to load the dataset in main memory (which I assume is the case considering what you explain), stochastic gradient descent is the first step you should take before trying dimensionality reduction techniques. With Scikit-learn on python (or R, but I'm not a user of this language), this would work just fine. $\endgroup$
    – Bertrand R
    Jun 3, 2014 at 14:18
  • $\begingroup$ I think this is a useful answer, but I think the OP is asking about the logistic regression, and not the feature reduction. Maybe you can address that part of the question in an edition? $\endgroup$
    – logc
    Jun 3, 2014 at 14:27
  • $\begingroup$ I'm not sure how useful PCA is for regression problems. The issue is this: PCA keeps the largest singular values of the input matrix, but the pseudo-inverse of the matrix inverts the singular values, so you really want to keep the smallest of the original values. It might be better to just sketch the data: arxiv.org/abs/1411.4357 $\endgroup$
    – Thomas Ahle
    Jul 25, 2019 at 19:40
4
$\begingroup$

As @Frank Harrell already mentioned, using elastic net or LASSO to perform penalized regression with all 5000 features (p) would be a good start for feature selection (one can't simply remove 3500 variables because they are not "statistically significant" with the dependent variable of interest). Either of these methods can be performed using the R package, glmnet.

In order to take into account the relationships shared between the potential predictor variables of interest (p = 5000), I would recommend running a random forest using the randomForest package and/or gradient boosting using the gbm package to assess the relative importance of the potential predictor variables in regards to the binary outcome. With this information, you will be much more prepared to build a more parsimonious logistic regression model.

Improve this answer
$\endgroup$
1
  • 3
    $\begingroup$ No, it is not correct to do data dredging to decide which parameters to remove from the model. The value of random forests, like elastic net, is that it incorporates the right amount of shrinkage. Starting over with a subset of the variables found in a way that was not masked to $Y$ will cause bias. $\endgroup$
    – Frank Harrell
    Jun 9, 2014 at 20:49
1
$\begingroup$

I assume you are not limited to R, since this is a big data problem you probably shouldn't be. You can try MLlib, which is Apache Spark's scalable machine learning library.

Apache Spark, in turn, is a fast and general engine for in-memory large-scale data processing. These operate on a Hadoop framework that allows for the distributed processing of large data sets across clusters of computers using simple programming models. It is designed to scale up from single servers to thousands of machines, each offering local computation and storage.

Note that 'thousands of machines' is optional(!), you can set it up on your local work/home desktop as well.

Going back to MLlib, it comes with the below algorithms out of the box:

  • K-means clustering with K-means|| initialization.
  • L1- and L2-regularized linear regression.
  • L1- and L2-regularized logistic regression.
  • Alternating least squares collaborative filtering, with explicit ratings or implicit feedback.
  • Naive Bayes multinomial classification.
  • Stochastic gradient descent.

If you are regularly working with big data, you may need to adopt a Hadoop solution.

Improve this answer
$\endgroup$
0
$\begingroup$

You can try Vowpal Wabbit: Vowpal Wabbit . It works well with very large datasets and very large number of features.

according to the website:

This is a project started at Yahoo! Research and continuing at Microsoft Research to design a fast, scalable, useful learning algorithm. VW is the essence of speed in machine learning, able to learn from terafeature datasets with ease. Via parallel learning, it can exceed the throughput of any single machine network interface when doing linear learning, a first amongst learning algorithms.

Improve this answer
$\endgroup$
0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.