I have a dataset of cars with price label as binary outcome including "affordable" and "costly". I aim to predict the whether a car is of "affordable" or "costly" using logistic regression model, with 80% of dataset as training set, and will predict the binary outcome later. But before training, I have so many independent variables (over 80 columns/independent variables) that I want to check which predictors are important for the model. My purpose is to choose only important predictors so that the prediction later is of best performance. My steps are as follows:

  1. Split into training and test set
  2. Build a "draft" logistic regression model using the training set and see in the result which predictors are statistically significant.
  3. Check the output of this model, for categorical variables such as "Vehicle_brand", (now in the result this variable is split into multiple vars because it is a factor), I use Wald test to see whether the Vehicle_brand as a whole is significant for the model. If significant, keep "Vehicle_brand" for fitting into training set later.
  4. Only retain the predictors that are significant in the first model to fit into training set the second time, then use this model to predict on test set.

label_index <- createDataPartition(y = cars_LR$price_label, p = 0.80, list = FALSE) trn_price <- cars_LR[label_index, ] tst_price<- cars_LR[-label_index, ]

#train the logistic regression model

trn_price_LR <- cars_LR %>%
  mutate(price_label = relevel(factor(price_label), ref =="affordable"))

"Draft" model price_LR_mod:

 price_LR_mod<-glm(price_label~., family=binomial, data=trn_price_LR)

#Wald test for some categorical variables

regTermTest(price_LR_mod, "Vehicle_brand")

#The output of my first "Draft" model is as below, since "affordable" is reference level, the result below is predict whether a car is more or less likely to be "costly" based on the sign of coefficients (if p-value is statistically significant) Call: glm(formula = price_label ~ ., family = binomial, data = trn_price_LR)

Deviance Residuals: 
    Min       1Q   Median       3Q      Max  
-4.0661  -0.0780  -0.0001   0.1010   3.9243  

Coefficients: (1 not defined because of singularities)
                                    Estimate Std. Error z value Pr(>|z|)


Vehicle_brandCitroA«n             -1.409e+00  3.706e-01  -3.801 0.000144 ***
Vehicle_brandCupra                 9.352e+00  4.864e+02   0.019 0.984660    
Vehicle_brandDacia                -4.307e+00  4.959e-01  -8.685  < 2e-16 ***
Vehicle_brandDaewoo               -1.454e+01  1.303e+03  -0.011 0.991095    
Vehicle_brandDaihatsu             -8.288e+00  2.767e+03  -0.003 0.997610    
Vehicle_brandDodge                -2.163e+00  5.396e-01  -4.008 6.12e-05 ***
Vehicle_brandDS Automobiles        1.634e+01  8.560e+02   0.019 0.984770    
Vehicle_brandFiat                 -1.836e+00  3.947e-01  -4.653 3.28e-06 ***
Vehicle_brandFord                 -9.328e-01  2.585e-01  -3.608 0.000308 ***
Vehicle_brandGMC                  -4.006e-02  3.859e+00  -0.010 0.991718    
Vehicle_brandHonda                 3.806e-01  4.010e-01   0.949 0.342552    
Vehicle_brandHummer                1.773e+01  3.956e+03   0.004 0.996425    
Vehicle_brandHyundai              -5.818e-01  3.581e-01  -1.625 0.104250    
Vehicle_brandInfiniti              9.909e-01  8.971e-01   1.105 0.269353    
Vehicle_brandInny                 -3.030e+00  4.845e+03  -0.001 0.999501    
Vehicle_brandIsuzu                -2.365e+00  3.791e+00  -0.624 0.532802    
Vehicle_brandJeep                  5.196e-02  4.865e-01   0.107 0.914953    
Vehicle_brandKia                  -7.480e-01  3.152e-01  -2.373 0.017630 *  
Vehicle_brandLada                 -3.118e+00  9.790e-01  -3.185 0.001447 ** 

ABS                                6.255e-01  3.111e-01   2.010 0.044384 *  
Electric_front_windows            -8.848e-01  2.435e-01  -3.634 0.000279 ***
Drivers_airbag                     2.691e-01  2.962e-01   0.909 0.363608    
Power_steering                     3.146e-01  1.696e-01   1.855 0.063569 .  
ASR_traction_control              -5.977e-02  1.247e-01  -0.479 0.631780    
Rear_view_camera                   8.393e-02  1.151e-01   0.729 0.466049    
Heated_side_mirrors                1.551e-01  1.134e-01   1.368 0.171430    
CD                                -3.767e-01  1.193e-01  -3.158 0.001591 ** 

Is my procedure okay? And if my way of choosing important variables is correct, how could I get the names of all significant predictors? For example I want to choose the predictors that are significant such as ABS, Electric_front_windows, Power_steering, CD etc. by coding rather than manually typing for my improved model?

  • $\begingroup$ Do you have any desire to do inference on the parameters? If not, why not throw them all in an elastic net? $\endgroup$
    – Dave
    Jun 6 '21 at 20:42
  • $\begingroup$ I am not familiar with elastic net, this is the first time I have heard it. The purpose of choosing important predictors is to remove some "useless" predictors. I want to keep important ones to better predict the price range later. By the way, could my procedure be considered acceptable? $\endgroup$
    – Jasmine N
    Jun 6 '21 at 20:48
  • 2
    $\begingroup$ This site has already many posts on feature selection with LR $\endgroup$ Jun 6 '21 at 20:53

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.