I am trying to find a set of good predictors using carets GA in R to train a few classification models. My dataset consists of around 4500 rows of 96 independent variables. I want to use GA to, ideally, reduce my predictors down to between 15 and 25.

  pop <- gafs_initial(vars = 20, popSize = 50)
  gafs_lrSelection(population = pop, fitness = 1:50)
  gafs_spCrossover(population = pop, fitness = 1:50, parents = 1:2)
  gacontrol<-gafsControl(functions = rfGA, method = "cv", number = 10, 
                         allowParallel = TRUE, verbose = TRUE, holdout = 0.15)
                   popSize = 50, iters = 10, pcrossover = 0.8, 
                   pmutation = 0.1, gafsControl = gacontrol, ntree = 250, importance = T)

After letting gafs run for almost 18! hours, I get the following.

Genetic Algorithm Feature Selection

4402 samples
96 predictors
5 classes: 'y0y1', 'y1y3', 'y3y5', 'y5y9', 'y9y12' 

Maximum generations: 10 
Population per generation: 50 
Crossover probability: 0.8 
Mutation probability: 0.1 
Elitism: 0 

Internal performance values: Accuracy, Kappa
Subset selection driven to maximize internal Accuracy 

External performance values: Accuracy, Kappa
Best iteration chose by maximizing external Accuracy 
External resampling method: Cross-Validated (10 fold) 
Subsampling for internal fitness calculation: 15%

During resampling:
  * the top 5 selected variables (out of a possible 96):
    FCR (100%), PTRR (100%), TOS (100%), TGN (100%), VST (100%)
  * on average, 68.9 variables were selected (min = 59, max = 82)

In the final search using the entire training set:
   * 77 features selected at iteration 7 including:
     AB, ABT, AGN, AGH_L, BON ... 
   * external performance at this iteration is

   Accuracy       Kappa 
     0.4919      0.2763 

Is there a way I can get the algorithm to select fewer variables? It doesn't matter if the accuracy takes a slight hit.

Side question: Using gafs and rfe with train uses much less time even though they are running in nested loops resulting in more folds. For example, with rfe going through 5 folds, using train via caretFuncs to train an rf model with 5 folds, took just 11-12 minutes. But running just rfe, with the default for functions(NULL) in rfeControl, and 10 folds, took ~9.5 hours. Is there something wrong with my method? My understanding is that the 5x5 folds should take much longer. The code is running on an i7-2720QM CPU @ 2.2 GHz with 8Gb ram.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.