9
$\begingroup$

I'm using the kernlab package in R to build an SVM for classifying some data.

The SVM is working nicely in that it provides 'predictions' of a decent accuracy, however my list of input variables is larger than I would like and I am unsure as to the relative importance of the different variables.

I'd like to implement a genetic-algorithm to select the sub-set of input variables that produces the best-trained/fittest SVM.

I'd like some help with choosing which R package to use when attempting this GA implementation (and possibly a brief psuedo-example).

I've looked a most of the R GA/P packages out there (RGP, genalg, subselect, GALGO), but I'm struggling conceptually to see how I would pass in my ksvm function as part of the fitness function and input my variable array as the population pool...?

Any help, thoughts, or nudges in the right direction gratefully received.

Thanks

code that solves this added below in a later EDIT

# Prediction function to be used for backtesting
pred1pd = function(t) {
print(t)
##add section to select the best variable set from those available using GA
  # evaluation function - selects the best indicators based on miminsied training error
mi.evaluate <- function(string=c()) {
    tmp <- data[(t-lookback):t,-1]
    x <- string
    tmp <- tmp[,x==1]
    tmp <- cbind(data[(t-lookback):t,1],tmp)
    colnames(tmp)[1] <- "targets"
    trainedmodel = ksvm(targets ~ ., data = tmp, type = ktype, kernel="rbfdot", kpar=list(sigma=0.1), C = C, prob.model = FALSE, cross = crossvalid)
    result <- error(trainedmodel)
    print(result)
    }

## monitor tge GA process
monitor <- function(obj) {
minEval = min(obj$evaluations);
plot(obj, type="hist");
}

## pass out the GA results; size is set to be the number of potential indicators
gaResults <- rbga.bin(size=39, mutationChance=0.10, zeroToOneRatio=10, evalFunc=mi.evaluate, verbose=TRUE, monitorFunc=monitor, popSize=50, iters=3, elitism=10)

## now need to pull out the best chromosome and rebuild the data frame based on these results so that we can train the model

bestChro <- gaResults$population[1,]
newData <- data[,-1]
newData <- newData[,bestChro==1]
newData <- cbind(data[,1],newData)
colnames(newData)[1] <- "targets"
print(colnames(newData))

# Train model using new data set
model = trainSVM(newData[(t-lookback):t, ], ktype, C, crossvalid)
# Prediction
pred = as.numeric(as.vector(predict(model, newData[t+1, -1], type="response")))
# Print for user inspection
print(pred)
}
$\endgroup$

2 Answers 2

7
$\begingroup$

My advice would be to not do this. The theoretical advantages of the SVM that avoid over-fitting apply only to the determination of the lagrange multipliers (the parameters of the model). As soon as you start performing feature selection, those advantages are essentially lost, as there is little theory that covers model selection or feature selection, and you are highly likely to over-fit the feature selection criterion, especially if you search really hard using a GA. If feature selection is important, I would use something like LASSO, LARS or Elastic net, where the feature selection arises via reguarisation, where the feature selection is more constrained, so there are fewer effective degrees of freedom, and less over-fitting.

Note a key advantage of the SVM is that is is an approximate implementation of a generalisation bound which is independent of the dimensionality of the feature space, which suggests that feature selection perhaps shouldn't necessarily be expected to improve performance, and if there is a defficiency in the selection prcess (e.g. over-fitting the selection criterion) it may well make things worse!

$\endgroup$
5
  • 5
    $\begingroup$ +1 A sweet puppy dies each time you do feature selection with genetic algorithms. $\endgroup$
    – user88
    Jun 28, 2012 at 15:50
  • $\begingroup$ @mbq LOL! (apparently I needed to type at least six more characters than I really wanted to.) $\endgroup$ Jun 29, 2012 at 8:54
  • 1
    $\begingroup$ @mbq puppies AND kittens it would appear, if my results are anything to go by... $\endgroup$
    – tfb
    Aug 21, 2012 at 11:17
  • $\begingroup$ @mbq I'm planning to do some feature selection with GAs for a paper I am writing at the moment (not expecting it to work, but some of the datasets have too many features for exhaustive search). Sorry Fido! $\endgroup$ Nov 7, 2014 at 19:04
  • $\begingroup$ @DikranMarsupial Well, I can only invite you to try pre-filtering them with some of my timber tools (; $\endgroup$
    – user88
    Nov 10, 2014 at 1:18
2
$\begingroup$

In the end I have ended up using the 'genalg' package on R. It means converting the winning chromosome from a binary format to represent the variables in my data, but this is relatively trivial once the GA has run. Let me know if you'd like any further details.

$\endgroup$
3
  • $\begingroup$ Would you mind posting the code? $\endgroup$
    – B_Miner
    Jul 16, 2012 at 1:54
  • $\begingroup$ @B_Miner sorry for the delay, it's a while since I've been on SO. It's also a while since I was struggling with this problem in R. I've had a look through my old files and I think the below was the code that solved it - hope it helps: added above $\endgroup$
    – tfb
    Aug 21, 2012 at 11:08
  • 2
    $\begingroup$ (Disclaimer: IMHO genetic algorithms are among the more evil optimizers for statistical models. They are exploiting variance in performance estimation in a very bad way). So: at least do check your final model with truly independent test data! $\endgroup$ Aug 21, 2012 at 15:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.