Bayesian Optimization for a Stochastic Target that changes over time Let's say there is a single slot machine that:


*

*costs zero to play

*can only be played once per day

*has a payout that is conditionally normal and is a function of the date and time.


I want to use Bayesian Optimization to maximize my total payout.
I have the following questions:


*

*Since the target is stochastic, are there any adjustments I need to make to my model/approach?

*How do I pick the parameters for the GP?

*Since the payout distribution changes over time, how do I take this into account? Do I use some sort of look back rolling window?


Thanks everyone!
 A: *

*You'll need an estimate of variance at each point. Either fitting an additional parameter $\sigma^2$ and using the kernel matrix $K+\sigma^2 I$ or achieving estimates of the variance of each point by using multiple draws, so that the effect is to give more weight to observations with lower variance.

*Standard methods include maximum likelihood, fully Bayesian inference and variations on cross-validation. The likelihood surface is typically nonconvex. My experience is that fully Bayesian methods tend to work the best because you're marginalizing the uncertainty in the hyperparameters and that MLEs methods tend to race to implausible hyperparameter values. More information is available in Rassmussen and Williams,  Gaussian Processes for Machine Learning. The stan documentation has illustrative fully Bayesian GP models.

*Maybe you could use a rolling window. But you should probably just encode the time-varying information in the kernel directly. The standard tricks are that the sums and Schur products of kernels are also kernels, so if you have one kernel encoding one kind of information, you can have another that encodes the time-varying information. The section of Gelman's Bayesian Data Analysis 3rd ed on Gaussian process methods includes an example of how to do so.
