I'm decently familiar with Gaussian Processes; I understand that GPs form the backbone of BO but I don't want this question to drift in scope towards an explanation of GPs. Rather, I'm curious, how does BO make decisions given a gaussian process as a surrogate function?
The GP models the problem parameters (to be optimized) as inputs, X, and the corresponding rewards as outputs, y, learning a smooth function that estimates rewards between adjacent observations (specific combinations of parameters to be optimized.) This is the surrogate function.
I can think of a few ways which a system might make parameter adjustments:
- Pick the parameter configuration corresponding to the absolute highest reward observed (Exploitation.)
- pick the highest variance region (few observations nearby) and observe what happens (Exploration.)
- Use reward and variance in some ranking scheme to decide where to move next.
Ex 1: A very small reward with extremely low variance- should be avoided.
Ex 2: A very high reward with very high variance- probably should be explored more to reduce variance around estimate.
Ex 3: A very high reward with very little variance- probably should not be explored further, however, may be the optimal parameter configuration.
Ex 4: Very low reward with very high variance- probably should be explored to reduce variance around estimate (but perhaps less so than in example 2?)
In answering this question, could you elaborate on how variance and reward are used to inform which regions of the surrogate function ought to be explored most vs least- what is this ranking metric/system?
Likewise, the term acquisition function appears to be very closely related to what I'm asking, but what is it and how does it guide exploration/exploitation?