I'm playing with tuning learning machines (specifically a random forest and a support vector machine) using genetic algorithms in R.
The only real complication that I've encountered is developing a good evaluation function. Specifically, I realized I wanted a function which would optimize the tuning parameters by reducing both bias and variance. Reducing bias was simple enough, but reducing variance proved a bit more tricky. In the end, I opted for the following evaluation function:
This yields the following 3D surface:
This is almost what I want. It maximizes when both errors are small (minimizes bias) and provides a continuous local maximum where both errors are equal (minimizes variance).
I'm not real experienced in inventing error surfaces, so I thought I'd post my initial efforts here to see if someone has some guidance or, perhaps, a better idea than me of what I'm trying to do.