I'm working on a behavioral scorecard modelling exercise, and many of the decisions taken to date have been based on the experience of a consulting credit analyst (whose experience software-wise is SAS) as I am primarily in BI. So far I have:

  • a linux pc with 32gb of ram and an i7 processor
  • an observation window
  • ~90 potential characteristics
  • a binary outcome

In R, I have

  1. loaded the dataset (225k obs of 88 vars, 1 outcome)
  2. split the dataset up based on the recommendations/examples in the package caret i.e. predictors and outcomes split up (150k obs in training sample)
  3. removed any variables showing a high degree of correlation (caret::findCorrelation)
  4. cut all continuous variables into categorical intervals
  5. reduced the number of variables based on near zero variance, missing values, and low information value (IV) (150k obs of 48 vars)
  6. tried bestglm::bestglm, caret::train (with glm and glmnet), FWDselect::selection, FWDselect::qselection but eventually had to interrupt each of these due to not completing after 4 hours of 100% CPU usage
  7. used FactoMineR:MCA to perform a multiple correspondence analysis (on predictors only)

What I would like to do is have a selection of logistic regression models for say 4, 8, 12, and 16 variables that are the most predictive models at each point. I'm not sure if I'm going in the correct direction here with MCA as I've mainly been simply trying to find something that works in a timely fashion for reducing my variables further or going directly to variable selection steps.

I would appreciate any advice on how to do any of step 6 better, whether 7 makes sense and what step 8 should be.

PS Design decisions up to 6&7 can't be revised so please, no telling me off for them!

  • 4
    $\begingroup$ #4. Seems like a silly idea if you have too many variables... This turns each continuous variable into a several categorical ones, which adds a lot more variables to your dataset. Try a simple logistic regression using glm and see how long it takes to fit. Also try taking a random sample of your dataset (maybe 10k rows) to test your modeling workflow. $\endgroup$ – Zach May 17 '13 at 15:36
  • $\begingroup$ Thank you for the suggestion of taking subset to try steps 6&7. With regards to 4 - as the purpose to have a 'scorecard', having the intervals which get assigned a score for each variable in the regression model is a necessity. $\endgroup$ – Steph Locke May 18 '13 at 13:02
  • $\begingroup$ FWDselect worked acceptably on a susbet of 20k, at this point would I need to do similarly on further subsets or would a feature selection based on a 10% sample be acceptable to move forward with? $\endgroup$ – Steph Locke May 19 '13 at 18:08
  • $\begingroup$ Maybe do it a few times with a 20k subset, and look how frequently each feature gets selected? $\endgroup$ – Zach May 19 '13 at 19:30
  • $\begingroup$ Cheers @Zach, I ran a different subset and got very different results. Someone recommend MASS::stepAIC to me, which I've been able to get running and working over the whole training subset - it's slow going but with the verbose output I'm able to identify at least that it's doing things and get an idea of how much longer I have left. $\endgroup$ – Steph Locke May 20 '13 at 10:50

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.