What is Gaussian in BO: Can I use bayesian optimization for binary classification task?

Question

BO works based on gaussian process, so what is gaussian there? Can I use BO for binary classification task, for example neural network which has output with logistic activation and is minimizing binary cross entropy loss ? target is thus from binary distribution and the noise is not gaussian.

Answer 1

Yes, the prior of objective function evaluation in your hyperparameter space is modelled using a Gaussian Process, the target need not be continuous, or have gaussian noise so it seems that it would be appropriate in your case.

This assumes some smoothness of your loss function in the hyperparameter space. Similar hyperparameters will have similar evaluations of the loss function, but does not require that the target be continuous, or have gaussian noise.

To give an example, imagine you are an oil exploration company choosing where to take samples on a square oilfield (X,Y). You might not start with any particular idea where there is lots of oil in the field, and so your expectation of the volume of oil at every point may be the same gaussian distribution.

If you start drilling randomly in the upper left area of the square, finding a reasonably large volume of oil, you might expect points nearby to also have a similar amount of oil, so your posterior belief for these points will have a higher mean and lower variance than points very far away in the lower right area. If you then checked a point in the lower right area (maybe the expected improvement from this could be higher because of the high variance) and found a very low amount of oil, then your posterior belief for the amount of oil in lower right area will now have a lower mean and variance.

Bayesian optimisation of hyperparameters is carrying out a similar process - X,Y represent the hyperparameters of your model and amount of oil found represents the evaluation of your loss function.

Here is a link going into more technical detail:

https://arxiv.org/abs/1807.02811

Here is some details of using BO to tune hyperparameters for a Keras NN

https://arxiv.org/pdf/1806.10282.pdf

Here is a code example of using BO on a classification task:

https://github.com/shibuiwilliam/keras_gpyopt/blob/master/bopt_nn.ipynb