# How to perform genetic-algorithm variable selection in R for SVM input variables?

I'm using the kernlab package in R to build an SVM for classifying some data.

The SVM is working nicely in that it provides 'predictions' of a decent accuracy, however my list of input variables is larger than I would like and I am unsure as to the relative importance of the different variables.

I'd like to implement a genetic-algorithm to select the sub-set of input variables that produces the best-trained/fittest SVM.

I'd like some help with choosing which R package to use when attempting this GA implementation (and possibly a brief psuedo-example).

I've looked a most of the R GA/P packages out there (RGP, genalg, subselect, GALGO), but I'm struggling conceptually to see how I would pass in my ksvm function as part of the fitness function and input my variable array as the population pool...?

Any help, thoughts, or nudges in the right direction gratefully received.

Thanks

code that solves this added below in a later EDIT

# Prediction function to be used for backtesting
pred1pd = function(t) {
print(t)
##add section to select the best variable set from those available using GA
# evaluation function - selects the best indicators based on miminsied training error
mi.evaluate <- function(string=c()) {
tmp <- data[(t-lookback):t,-1]
x <- string
tmp <- tmp[,x==1]
tmp <- cbind(data[(t-lookback):t,1],tmp)
colnames(tmp)[1] <- "targets"
trainedmodel = ksvm(targets ~ ., data = tmp, type = ktype, kernel="rbfdot", kpar=list(sigma=0.1), C = C, prob.model = FALSE, cross = crossvalid)
result <- error(trainedmodel)
print(result)
}

## monitor tge GA process
monitor <- function(obj) {
minEval = min(obj$evaluations); plot(obj, type="hist"); } ## pass out the GA results; size is set to be the number of potential indicators gaResults <- rbga.bin(size=39, mutationChance=0.10, zeroToOneRatio=10, evalFunc=mi.evaluate, verbose=TRUE, monitorFunc=monitor, popSize=50, iters=3, elitism=10) ## now need to pull out the best chromosome and rebuild the data frame based on these results so that we can train the model bestChro <- gaResults$population[1,]
newData <- data[,-1]
newData <- newData[,bestChro==1]
newData <- cbind(data[,1],newData)
colnames(newData)[1] <- "targets"
print(colnames(newData))

# Train model using new data set
model = trainSVM(newData[(t-lookback):t, ], ktype, C, crossvalid)
# Prediction
pred = as.numeric(as.vector(predict(model, newData[t+1, -1], type="response")))
# Print for user inspection
print(pred)
}

-

My advice would be to not do this. The theoretical advantages of the SVM that avoid over-fitting apply only to the determination of the lagrange multipliers (the parameters of the model). As soon as you start performing feature selection, those advantages are essentially lost, as there is little theory that covers model selection or feature selection, and you are highly likely to over-fit the feature selection criterion, especially if you search really hard using a GA. If feature selection is important, I would use something like LASSO, LARS or Elastic net, where the feature selection arises via reguarisation, where the feature selection is more constrained, so there are fewer effective degrees of freedom, and less over-fitting.

Note a key advantage of the SVM is that is is an approximate implementation of a generalisation bound which is independent of the dimensionality of the feature space, which suggests that feature selection perhaps shouldn't necessarily be expected to improve performance, and if there is a defficiency in the selection prcess (e.g. over-fitting the selection criterion) it may well make things worse!

-
+1 A sweet puppy dies each time you do feature selection with genetic algorithms. –  mbq Jun 28 '12 at 15:50
@mbq LOL! (apparently I needed to type at least six more characters than I really wanted to.) –  Dikran Marsupial Jun 29 '12 at 8:54
@mbq puppies AND kittens it would appear, if my results are anything to go by... –  tfb Aug 21 '12 at 11:17
@mbq I'm planning to do some feature selection with GAs for a paper I am writing at the moment (not expecting it to work, but some of the datasets have too many features for exhaustive search). Sorry Fido! –  Dikran Marsupial Nov 7 at 19:04
@DikranMarsupial Well, I can only invite you to try pre-filtering them with some of my timber tools (; –  mbq Nov 10 at 1:18

In the end I have ended up using the 'genalg' package on R. It means converting the winning chromosome from a binary format to represent the variables in my data, but this is relatively trivial once the GA has run. Let me know if you'd like any further details.

-
Would you mind posting the code? –  B_Miner Jul 16 '12 at 1:54
@B_Miner sorry for the delay, it's a while since I've been on SO. It's also a while since I was struggling with this problem in R. I've had a look through my old files and I think the below was the code that solved it - hope it helps: added above –  tfb Aug 21 '12 at 11:08
(Disclaimer: IMHO genetic algorithms are among the more evil optimizers for statistical models. They are exploiting variance in performance estimation in a very bad way). So: at least do check your final model with truly independent test data! –  cbeleites Aug 21 '12 at 15:06