A continuous generalization of the binary bandit There is plenty of reading out there about Bayesian (beta-binomial) multiarm bandits for 0/1 data, but I would like to extend this slightly. 
To give some context, suppose I have two webpages, A and B. Now I want to test which webpage gets more people to call in, so I begin by randomly serving A, B to incoming visitors. 
This is equivalent to starting with the same beta priors on A, B - either $\text{prior}_A = \text{prior}_B = \mathrm{Beta}(0,0)$ aka uninformed or some $\mathrm{Beta}(k_1, k_2)$ for both. 
As the binary data comes in (either people call or they don't), I update according to Bayes' Rule and so I end up with $\text{posterior}_A = \mathrm{Beta}(k_1 + \text{wins}_A, k_2 + \text{trials}_A - \text{wins}_A)$ and similarly for B. 
Here a trial is me serving the visitor a webpage and a success is a call, to be clear. 
My Question
What is the continuous analogue of this binary data model? That is, if my data coming in is now of the form $0.5, 1.6, 8.95$ etc., what can I use to optimize which is better A or B? I have looked into Gaussian processes a little bit but I'm not sure this is what I want. Thanks for any help. 
An extension to this question may be found here.
 A: Disclosure: I know almost nothing about bandits. Still, my suggestion seems like a natuaral generalization of the case you presented. It does not consider the experimental design step in detail (since I don't know what people ususally consider as a loss function in this scenario), so might fail in this respect.
Let me ignore the fact that we have two web pages. You have a web page and your prior for "profit per call" (denoted $\theta$) is (say) Gaussian- $p(\theta) = (2\pi)^{-\frac{1}{2}} \exp( -\frac{\theta^2}{2})$. Lets also say that you believe that if you know the average profit per view, your gains are distributed according to another normal $X \sim \mathcal{N}(\theta, 1)$. Then you have a valid (and very simple) bayesian model.
Now, use the same model for both web pages. After observing $n$ (potential) clients' reactions, you have the distributions for the posterior of $A$ and $B$ which are both Gaussian. Based on whatever target you choose for the experimental design step, you can choose which web page to present next. Calculating and differentiating the target should not be too hard, since we know how to integrate Gaussians. 
A: This problem is tackled by the Dearden paper on Bayesian Q-Learning.  He considers a normal model for "returns" (future rewards) like @yair's solution.  He also considers another term called the "value of information": "the expected improvement in future decision quality that might arise from the information acquired by exploration."
Also, FYI Beta(0, 0) is not "uninformative".  The Jeffreys prior is Beta(0.5, 0.5).
Dearden, Richard, Nir Friedman, and Stuart Russell. "Bayesian Q-learning." AAAI/IAAI. 1998.
