The setting you describe resembles non-stationary bandits, in which the means of the arms vary over time. 

The following papers might be of interest:

1. [A Linear Bandit for Seasonal Environments][1]
2. [Stochastic Multi-Armed-Bandit Problem with Non-stationary Rewards][2]
3. [On Slowly-varying Non-stationary Bandits][3]

Here there are also have these [lecture notes][4] which seem relevant.


  [1]: https://arxiv.org/pdf/2004.13576.pdf
  [2]: https://proceedings.neurips.cc/paper/2014/file/903ce9225fca3e988c2af215d4e544d3-Paper.pdf
  [3]: https://arxiv.org/abs/2110.12916
  [4]: https://courses.cs.washington.edu/courses/cse599m/21sp/resources/week9_scribe.pdf