Adapting UCB1 algorithm for contextual bandits I am using the UCB1 algorithm to solve the multi-armed bandit problem and I would like to adapt it to handle some context vector that would influence the UCB algorithm that makes recommendations. Is there a simple way to do this? 
 A: The algorithm that fits that description most closely would be LinUCB, as described in Algorithm 1 of this paper.
Basically, the algorithm as described there assumes that in every round $t$, for every arm $a$, you are presented with a feature vector $\mathbf{x}_{t, a}$. For every arm $a$, it tries to learn a vector of parameters $\hat{\boldsymbol{\theta}}_a$, such that the dot product $\hat{\boldsymbol{\theta}}_a^{\top} \mathbf{x}_{t, a}$ between those two vectors estimates the reward that you expect to get from playing arm $a$ at time $t$. 
That expression $\hat{\boldsymbol{\theta}}_a^{\top} \mathbf{x}_{t, a}$ is comparable to the term for the average reward of an arm in the non-contextual UCB1 algorithm (the "exploitation" term). In the paper's pseudocode, you see this expression appearing in line 9, which is basically computing the "Upper Confidence Bounds" in the contextual/linear setting. 
The other expression in that line is $\alpha \sqrt{\mathbf{x}_{t, a}^{\top} \mathbf{A}_a^{-1} \mathbf{x}_{t, a}}$. This expression can intuitively be understood as the part that computes the gap between the mean/expected/predicted reward, and the Upper Confidence Bound (the "exploration term"). This is the part that should make sure that you will occasionally play an arm that does not have the highest expected reward, just for the sake of exploration. The $\alpha$ parameter is an exploration parameter (high value means lots of exploration, low values means primarily exploitation). In the non-contextual, standard version of UCB1, a similar parameter is often denoted by $C$ or simply assumed to have a constant value of $\sqrt{2}$.
The parameters $\hat{\boldsymbol{\theta}}_a$ are basically computed using an incremental variant of Ridge Regression (a variant of Linear Regression). An incremental variant is used because, in MAB problems, you generally want to learn after every time step rather than collecting a whole bunch of data and then learning at once.
Note that the algorithm involves dot products, matrix multiplications and matrix inverses. If you want to implement it, you'll likely want to make sure that you have access to a good library for linear algebra (e.g. numpy in the case of python).
For a different explanation and perhaps slightly more clear pseudocode, it looks like this could be a good blog post (didn't extensively read it though, just found it through google).

Note that there are also settings where you only have a single feature vector $\mathbf{x}_t$ per time step $t$, rather than one feature vector per arm per time step. In such a case, the same algorithm can be used, you'll just want to make sure that you then also only learn a single vector of parameters $\hat{\boldsymbol{\theta}}$, rather than one per arm.
