Suppose we have a list of items $I_1,\cdots,I_K$, and we would like to order them by popularity using a Multi Armed Bandit approach. As a concrete example, imagine we're trying to advertise a toy on the internet and each $I_k$ is an image of the toy, so that we're trying to figure out which image has the highest click-thru rate. Since the click-thru rate is extremely low for all such advertisements, it becomes really expensive and long to figure out which image performs best.
Suppose now we have a deep learning model $P$ that predicts imagine popularity (based on prior data). We train this model and are now able to get popularity predictions (i.e. click-thru rate) $P(I_k)$ for each image. The model is far from perfect, but we do end up observing some correlation $C$ between our predictions and the true popularity.
I would like to augment this model with a multi-armed bandit to figure out which image is the most popular. The setup is a list of images $I_k$, along with a list of priors $P(I_k)$, with a correlation coefficient $C$ between the priors and the true popularity.
I'm not an expert in this area, so I was hoping to get some advice on what the exact augmentation procedure should be. As an example, reading this led me to Boltzmann Exploration, which generalizes the epsilon-greedy method to selecting arms based on their empirical means. In this case, we select image $i$ to test on at time $t+1$ based on:
$$p_i(t+1)=\frac{e^{\mu_i(t)/\tau}}{\sum_{i=1}^Ke^{\mu_i(t)/\tau}},$$
where $\mu_i(t)$ are the empirical means observed at time $t$ and $\tau$ is hyperparameter that controls for how uniformly random (or singular) we want the predictions to be. With the above neural network output, the easiest thing I can think of is replacing $\mu_i(t)\rightarrow \mu_i(t)+P(I_i)$. Are there better approaches? I would sincerely appreciate any literature references on this.