I am trying to build a multiple regression model where I have 7 independent variables(predictors), out of which 3 are numerical and 4 are categorical(with each of factor levels upto 80). I need to filter the best possible predictors(variable selection) from these. Is there any way I can do this? I read about Lasso but I guess it can be applied only if all the predictors are of numerical in nature. Step wise selection is generally not advisable. Please help me with your ideas. Thanks.

  • 1
    $\begingroup$ Standard lasso regression libraries do handle factors directly. You would have a problem if you don't have enough examples to train with(as you have a lot of regressors). If that's the case, correlation with the independent variables should help you figure out what should be good predictors. $\endgroup$ – Ujjwal Kumar Jan 10 '17 at 10:30
  • $\begingroup$ @UjjwalKumar: Thanks. Do you mean the lars and glmnet packages? If so, from that how do I select my important variables? $\endgroup$ – Maharajan N Jan 10 '17 at 10:53
  • 1
    $\begingroup$ You don't need to, the packages should automatically do that for you. Variables with high coefficients should be important ones $\endgroup$ – Ujjwal Kumar Jan 10 '17 at 10:57
  • $\begingroup$ @UjjwalKumar I don't agree that "high coefficients" should be a selection criteria for variable importance for the simple reason that coefficients are expressed in the units of x, the predictor or feature. Using that rule, continuous predictors with high average (or median) values and contingently large std deviations would always be preferred over predictors with small values and std deviations. Similarly, categorical predictors with many buckets would likely be preferred over categorical predictors with only a few buckets. One heuristic is to prefer predictors with high F or t-statistics $\endgroup$ – Mike Hunter Jan 10 '17 at 13:15

Decision trees and Random forests are known to be useful to do feature selection. If you train your data on any of those at the end you'll be able to sort features by importante. The nature of those methods works by kind of splitting data based on the gain of information when a particular feature is used to split data. If you split the data with a good feature you'll will gain more information than if you split data with a bad feature.

Check this reference using R on the examples: http://freakonometrics.hypotheses.org/19835

The thing with lasso is that l1 regularization can produce 0 coefficients so people often refers to that property as a 'builtin feature selector'. If the coefficient is 0 it means that this feature it's not relevant to describe the data. It's also a popular way to do feature selection.

  • $\begingroup$ Those algorithms are really slow when I have variables with factor levels upto 80. For example random forest does not work here. It throws error stating "Error in randomForest.default(m, y, ...)Can not handle categorical predictors with more than 53 categories. $\endgroup$ – Maharajan N Jan 10 '17 at 13:22
  • $\begingroup$ You can reduce the dimensionality of your data using PCA (or another dimensionality reduction algorithm) and then run a RF on a smaller set of features. $\endgroup$ – Felipe Cruz Jan 10 '17 at 13:38
  • $\begingroup$ Thanks. But I am unfamiliar with PCA and other dimensionality reduction algorithms :( $\endgroup$ – Maharajan N Jan 10 '17 at 13:44
  • $\begingroup$ Check this: stats.stackexchange.com/questions/57467/… $\endgroup$ – Felipe Cruz Jan 10 '17 at 13:48
  • 1
    $\begingroup$ @TommasoGuerrini Right. PCA was a suggestion without too much thought. But there are other possibilities to reduce the dimensionality of data from the original question: stats.stackexchange.com/questions/5774/… $\endgroup$ – Felipe Cruz Jan 10 '17 at 14:52

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.