Using genetic algorithm to tune learning machines

I'm playing with tuning learning machines (specifically a random forest and a support vector machine) using genetic algorithms in R.

The only real complication that I've encountered is developing a good evaluation function. Specifically, I realized I wanted a function which would optimize the tuning parameters by reducing both bias and variance. Reducing bias was simple enough, but reducing variance proved a bit more tricky. In the end, I opted for the following evaluation function:

$\frac{1}{\left&space;|&space;train.rmse&space;-&space;test.rmse&space;\right&space;|+\left&space;(&space;train.rmse&space;+&space;test.rmse&space;\right&space;)}$

This yields the following 3D surface:

This is almost what I want. It maximizes when both errors are small (minimizes bias) and provides a continuous local maximum where both errors are equal (minimizes variance).

I'm not real experienced in inventing error surfaces, so I thought I'd post my initial efforts here to see if someone has some guidance or, perhaps, a better idea than me of what I'm trying to do.

• your error surface aside : ooc: in what sense is this a genetic algorithm? – javadba Oct 9 '15 at 20:12
• I'm addressing the evaluation function only, here. The genetic algorithm package I'm using is the GA library available on CRAN. I'm really only concerned with whether or not I can encourage the GA's ability to find an optimal solution more easily by engineering where local extrema occur in the error surface that the evaluation function provides. – Joel Graff Oct 10 '15 at 10:47
• OK - I re-read your OP and it is relatively clear on that. Interesting question. – javadba Oct 10 '15 at 15:26

In terms of objective function: training performance is usually entirely irrelevant. Most objective functions work based on a cross-validation estimate of some performance metric (e.g. MSE, $R^2$, AUROC, ...). Optunity's cross-validation features allow you to compute every statistic you might want based on cross-validation, so you could incorporate variance if you'd like.