I am using Random Forest and Stochastic Gradient Boosting to predict a categorical target variable exhibiting severe between-class imbalance. I am using oversampling to make sure that the models do not neglect the minority classes during training and reach decent test-set accuracy for all classes.

As recommended by the literature (e.g., Sun et al. (2007)), I am considering the oversampling proportions as hyperparameters requiring optimization. Finding the best number of observations to sample from each class translates to an optimization problem in a 6-dimensional (pseudo)-continuous space (I have 6 classes).

So, far, I have used manual trial and error: I initially sample the same number of observations from each class, then assess error rates for each category using cross-validation, and increase/decrease the number of observations drawn from each class until the error is equally shared among all categories.

But I would like to know if it's possible to automate this optimization procedure. Should I use as an optimizer gradient descent/ascent, or a genetic algorithm?

An exhaustive grid search would be way too compute intensive, even with a rough grid, and I know simulated annealing is usually only used in discrete spaces.

Could someone familiar with similar optimization problems please point me towards a good solution? I am not very used to optimizers different from grid search, and I don't want to be reinventing the wheel.

  • $\begingroup$ Not entirely sure what you are asking. Are you looking for optimization algorithms for the training problems (e.g. QP for SVM), or are you looking for hyperparameter optimization methods (e.g. optimizing C and kernel parameters for SVM)? Grid search and gradient descent, for instance, are never both viable options for a certain problem. $\endgroup$ – Marc Claesen Aug 4 '15 at 15:47
  • $\begingroup$ Yes, my question deals with hyperparameter optimization algorithms $\endgroup$ – Antoine Aug 4 '15 at 20:32
  • $\begingroup$ edited question to make it clear $\endgroup$ – Antoine Aug 4 '15 at 20:41
  • 1
    $\begingroup$ I think this question is far too broad: it's basically asking to compare and contrast the relative merits of 5 (or more!) optimization methods. A typical graduate-level applied math curriculum would teach an entire course on this topic! $\endgroup$ – Sycorax Aug 4 '15 at 22:00
  • 1
    $\begingroup$ Oh, and it's not just comparing/contrasting five algorithms, but comparing and contrasting them for SVM, RF and any other ML classifier, out of concern that decent optimizers for one ML algorithm might be suboptimal for another. $\endgroup$ – Sycorax Aug 4 '15 at 22:07

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.