# Bayesian Optimization for Non-Stationary, Contextual, Infinitely-many Armed Bandits

What algorithms are known to work well with contextual, non-stationary, infinite armed bandits?

In case it isn't clear:

• Contextual: I mean that the rewards depend, not only on our actions, but state (e.g. a feature vector) as well.
• Non-stationary Rewards $R$ are also a function of time, i.e. $R(s,a,t)$
• Infinitely-many arms: A subset of the arms live in $\rm I\!{R}^d$ (e.g. say $d$ arms live in $\rm I\!{R}$, i.e. a continuum).

Bayesian optimization methods are particularly relevant here, but, if I understand correctly, most solutions I have seen don't consider state or non-stationarity .

When it comes to factoring in state, one thought I had to was to include it as part of the data on which we condition the posterior. That is, the Gaussian Process kernel would specify covariance not only on dimensions of the arms, but also on the state, even if the state is always known (but varying of course).

Would doing the latter make sense? If so, what's the advice on these types of kernels? Any other way to accommodate state?

Also, how do we accommodate non-stationarity in the model? Perhaps using an exponential kernel to model time? If we go down this route, how does one combine all these kernels into a single GP?

• Could you make it more specific what do you mean by "continuum armed bandits"? What are the arms in this scenario?
– Tim
Nov 5, 2016 at 20:46
• Thanks @Tim I updated the OP. Let me know if it isn't still clear. Nov 5, 2016 at 21:07