3
$\begingroup$

I'm playing with tuning learning machines (specifically a random forest and a support vector machine) using genetic algorithms in R.

The only real complication that I've encountered is developing a good evaluation function. Specifically, I realized I wanted a function which would optimize the tuning parameters by reducing both bias and variance. Reducing bias was simple enough, but reducing variance proved a bit more tricky. In the end, I opted for the following evaluation function:

This yields the following 3D surface:

enter image description here

This is almost what I want. It maximizes when both errors are small (minimizes bias) and provides a continuous local maximum where both errors are equal (minimizes variance).

I'm not real experienced in inventing error surfaces, so I thought I'd post my initial efforts here to see if someone has some guidance or, perhaps, a better idea than me of what I'm trying to do.

$\endgroup$
  • $\begingroup$ your error surface aside : ooc: in what sense is this a genetic algorithm? $\endgroup$ – javadba Oct 9 '15 at 20:12
  • $\begingroup$ I'm addressing the evaluation function only, here. The genetic algorithm package I'm using is the GA library available on CRAN. I'm really only concerned with whether or not I can encourage the GA's ability to find an optimal solution more easily by engineering where local extrema occur in the error surface that the evaluation function provides. $\endgroup$ – Joel Graff Oct 10 '15 at 10:47
  • $\begingroup$ OK - I re-read your OP and it is relatively clear on that. Interesting question. $\endgroup$ – javadba Oct 10 '15 at 15:26
2
$\begingroup$

If you want to experiment with meta-heuristics for hyperparameter optimization, I strongly suggest having a look at Optunity (documentation and paper), which is an open source library that does exactly that. The default optimizer is particle swarm optimization, which we have found to perform very well on a large variety of problems, but we also offer the covariance matrix adaptation evolutionary strategy.

In terms of objective function: training performance is usually entirely irrelevant. Most objective functions work based on a cross-validation estimate of some performance metric (e.g. MSE, $R^2$, AUROC, ...). Optunity's cross-validation features allow you to compute every statistic you might want based on cross-validation, so you could incorporate variance if you'd like.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.