1
$\begingroup$

I am trying to find a set of good predictors using carets GA in R to train a few classification models. My dataset consists of around 4500 rows of 96 independent variables. I want to use GA to, ideally, reduce my predictors down to between 15 and 25.

  require(doParallel)
  cl<-makeCluster(detectCores()-1)
  registerDoParallel(cl)
  set.seed(5151)
  pop <- gafs_initial(vars = 20, popSize = 50)
  gafs_lrSelection(population = pop, fitness = 1:50)
  gafs_spCrossover(population = pop, fitness = 1:50, parents = 1:2)
  gacontrol<-gafsControl(functions = rfGA, method = "cv", number = 10, 
                         allowParallel = TRUE, verbose = TRUE, holdout = 0.15)
  gafs.train<-gafs(hdtrain[,-ncol(hdtrain)],hdtrain[,ncol(hdtrain)], 
                   popSize = 50, iters = 10, pcrossover = 0.8, 
                   pmutation = 0.1, gafsControl = gacontrol, ntree = 250, importance = T)
  stopCluster(cl)

After letting gafs run for almost 18! hours, I get the following.

Genetic Algorithm Feature Selection

4402 samples
96 predictors
5 classes: 'y0y1', 'y1y3', 'y3y5', 'y5y9', 'y9y12' 

Maximum generations: 10 
Population per generation: 50 
Crossover probability: 0.8 
Mutation probability: 0.1 
Elitism: 0 

Internal performance values: Accuracy, Kappa
Subset selection driven to maximize internal Accuracy 

External performance values: Accuracy, Kappa
Best iteration chose by maximizing external Accuracy 
External resampling method: Cross-Validated (10 fold) 
Subsampling for internal fitness calculation: 15%

During resampling:
  * the top 5 selected variables (out of a possible 96):
    FCR (100%), PTRR (100%), TOS (100%), TGN (100%), VST (100%)
  * on average, 68.9 variables were selected (min = 59, max = 82)

In the final search using the entire training set:
   * 77 features selected at iteration 7 including:
     AB, ABT, AGN, AGH_L, BON ... 
   * external performance at this iteration is

   Accuracy       Kappa 
     0.4919      0.2763 

Is there a way I can get the algorithm to select fewer variables? It doesn't matter if the accuracy takes a slight hit.

Side question: Using gafs and rfe with train uses much less time even though they are running in nested loops resulting in more folds. For example, with rfe going through 5 folds, using train via caretFuncs to train an rf model with 5 folds, took just 11-12 minutes. But running just rfe, with the default for functions(NULL) in rfeControl, and 10 folds, took ~9.5 hours. Is there something wrong with my method? My understanding is that the 5x5 folds should take much longer. The code is running on an i7-2720QM CPU @ 2.2 GHz with 8Gb ram.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.