I am trying to find a set of good predictors using carets GA in R to train a few classification models. My dataset consists of around 4500 rows of 96 independent variables. I want to use GA to, ideally, reduce my predictors down to between 15 and 25.
require(doParallel) cl<-makeCluster(detectCores()-1) registerDoParallel(cl) set.seed(5151) pop <- gafs_initial(vars = 20, popSize = 50) gafs_lrSelection(population = pop, fitness = 1:50) gafs_spCrossover(population = pop, fitness = 1:50, parents = 1:2) gacontrol<-gafsControl(functions = rfGA, method = "cv", number = 10, allowParallel = TRUE, verbose = TRUE, holdout = 0.15) gafs.train<-gafs(hdtrain[,-ncol(hdtrain)],hdtrain[,ncol(hdtrain)], popSize = 50, iters = 10, pcrossover = 0.8, pmutation = 0.1, gafsControl = gacontrol, ntree = 250, importance = T) stopCluster(cl)
gafs run for almost 18! hours, I get the following.
Genetic Algorithm Feature Selection 4402 samples 96 predictors 5 classes: 'y0y1', 'y1y3', 'y3y5', 'y5y9', 'y9y12' Maximum generations: 10 Population per generation: 50 Crossover probability: 0.8 Mutation probability: 0.1 Elitism: 0 Internal performance values: Accuracy, Kappa Subset selection driven to maximize internal Accuracy External performance values: Accuracy, Kappa Best iteration chose by maximizing external Accuracy External resampling method: Cross-Validated (10 fold) Subsampling for internal fitness calculation: 15% During resampling: * the top 5 selected variables (out of a possible 96): FCR (100%), PTRR (100%), TOS (100%), TGN (100%), VST (100%) * on average, 68.9 variables were selected (min = 59, max = 82) In the final search using the entire training set: * 77 features selected at iteration 7 including: AB, ABT, AGN, AGH_L, BON ... * external performance at this iteration is Accuracy Kappa 0.4919 0.2763
Is there a way I can get the algorithm to select fewer variables? It doesn't matter if the accuracy takes a slight hit.
Side question: Using
train uses much less time even though they are running in nested loops resulting in more folds. For example, with
rfe going through 5 folds, using
caretFuncs to train an
rf model with 5 folds, took just 11-12 minutes. But running just
rfe, with the default for
rfeControl, and 10 folds, took ~9.5 hours. Is there something wrong with my method? My understanding is that the 5x5 folds should take much longer. The code is running on an i7-2720QM CPU @ 2.2 GHz with 8Gb ram.