Selecting Bayesian priors for the dependent data

I have goal of measuring CTRs of several titles of an article on a website using Bayesian approach.

In a simple setup, what one will do is to select Beta Prior (for example Uniform distribution) and run series of experiments until he gets credible intervals narrow enough to select proper title.

Let's assume now that I run the experiment on several websites simultaneously and want to understand statistics regarding each title-website pair - what should I do in this case?

Again, the simplest way is just to treat all pairs as independent and reuse approach from first phase.

What is however the best way to use the shared information, like number of trials and successes of particular title on all websites, while doing the experiment? It seems that this can lead to the credible results much faster (i.e. maybe one site is just not converting at all as there only bots there, or one title is performing much better on all websites)

What is the best way to update posterior distributions (or to reselect priors) using shared information between the entities?

Data Example

site_id title_id impressions clicks
123     t1       10051       150
123     t2       560         1
.
.
.
789     t1       101         15
789     t2       1050        2
.

• Sounds like you are after a hierarchical Bayesian model. Could you tell us more about your data (e.g. give a small example), so that we can understand it better? – Tim Nov 7 '17 at 21:52
• I've added data example by @tim 's request – Sergio Kozlov Nov 8 '17 at 9:08

\begin{align} \alpha \sim \mathcal{N}(0, \sigma^2_\alpha) \qquad \beta_i &\sim \mathcal{N}(\mu_\beta, \sigma^2_\beta) \qquad \gamma_j \sim \mathcal{N}(\mu_\gamma, \sigma^2_\gamma) \\ p_{ij} &= \mathrm{logit}^{-1}(\alpha + \beta_i + \gamma_j) \\ \mathsf{clicks}_{~ij} &\sim \mathcal{Bin}(p_{ij},\; n_{ij}) \end{align}
where $n_{ij}$ are the number of impressions (if I understand your data correctly), with some additional hyperpriors on $\sigma^2_\alpha, \sigma^2_\beta, \sigma^2_\gamma, \mu_\alpha, \mu_\beta$. As you can notice, this is a variation of Bayesian logistic regression model with "random effects" for each site $\beta_i$ and for each title $\gamma_j$. This lets you model the impact of individual sites and titles on views. Moreover, you can make use of all the data you have. Additionally, you can include some other explanatory variables in your model as well (in the regression part), what makes it very flexible.