Please apologize my butchering of statistical lingo :) I have found a couple of questions on here that are related to advertising and click through rates. But none of them helped me very much with my understanding of my hierarchical situation.
There's a related question Are these equivalent representations of the same hierarchical Bayesian model?, but I'm not sure if they actually have a similar problem. Another question Priors for hierarchial Bayesian binomial model goes into detail about hyperpriors, but I'm not able to map their solution to my problem
I have a couple of ads online for a new product. I let the ads run for a couple of days. At that point enough people have clicked on the ads to see which one gets the most clicks. After kicking out all but the one that has the most clicks I let that one run for another couple of days to see how much people actually purchase after clicking on the ad. At that point I know if it was a good idea to run the ads in the first place.
My statistics are very noisy because I don't have a lot of data since I'm only selling a couple of items every day. Therefore it is really hard to estimate how many people buy something after seeing an ad. Only about one in every 150 clicks results in a purchase.
Generally speaking I need to know if I'm losing money on each ad as soon as possible by somehow smoothing the per-ad group statistics with global statistics over all ads.
- If I wait until every ad has seen enough purchases I'll go broke because it takes too long: testing 10 ads I need to spend 10 times more money so that the statistics for each ad gets reliable enough. By that time I might have lost money.
- If I average purchases over all of the ads I won't be able to kick out ads that just aren't working as well.
Could I use the global purchase rate ($ per click) and use it as a prior for $N$ sub-distributions? That would mean that the more data I have for each ad, the more independent the statistics for that ad get. If nobody has clicked on an ad yet, I assume that the global average is appropriate.
Which distribution would I choose for that?
If I have had 20 clicks on A and 4 clicks on B, how can I model that? So far I have figured out that a binomial or Poisson distribution might make sense here:
purchase_rate ~ poisson
(?)(purchase_rate | group A) ~ poisson
(estimate the purchase rate only for the group A?)
But what do I do next to actually calculate the purchase_rate | group A
. How do I plug two distributions together to make sense for group A (or any other group).
Do I have to fit a model first? I have data that I could use to "train" a model:
- Ad A: 352 clicks, 5 purchases
- Ad B: 15 clicks, 0 purchases
- Ad C: 3519 clicks, 130 purchases
I'm looking for a way to estimate the probability of any one of the groups. If a group has only a couple of datapoints I essentially want to fall back to the global average. I know a bit about Bayesian statistics and have read a lot of PDFs of people describing how they model using Bayesian inference and conjugate priors and so on. I think there is a way to do this properly but I can't figure out how to model it correctly.
I would be super happy about hints that help me formulate my problem in a Bayesian way. That would help a lot with finding examples online that I could use to actually implement this.
Update:
Thanks so much for responding. I'm beginning to understand more and more little bits about my problem. Thank you! Let me ask a few questions to see if I understand the problem a bit better now:
So I assume the conversions are are distributed as Beta-distributions, and a Beta distribution has two parameters, $a$ and $b$.
The $\frac{1}{2}$ $\frac{1}{2}$ parameters are hyperparameters, so they are parameters to the prior? So in the end I set the number of conversions and number of clicks as the parameter of my Beta distribution?
At some point when I want to compare different ads, so I would compute $P(\mathrm{conversion} | \mathrm{ad}=X) = \frac{P(\mathrm{ad}=X | \mathrm{conversion}) * P(\mathrm{conversion})}{P(\mathrm{ad}=X)}$. How do I compute each part of that formula?
I think $P(\mathrm{ad}=X | \mathrm{conversion})$ is called likelihood, or "mode" of the Beta distribution. So that's $\frac{\alpha - 1}{\alpha + \beta - 2}$, with $\alpha$ and $\beta$ being the parameters of my distribution. But the specific $\alpha$ and $\beta$ here are the parameters for the distribution just for the ad $X$, right? In that case, is it just the number of clicks and conversions this ad has seen? Or is how many clicks/conversions all ads have seen?
Then I multiply with the prior, which is P(conversion), which is in my case just the Jeffreys prior, which is non-informative. Will the prior stay the same as I get more data?
I divide by $P(\mathrm{ad})$, which is the marginal likelihood, so I count how often this ad has been clicked?
In using Jeffreys' prior, I'm assuming that I'm starting at zero and don't know anything about my data. That prior is called "non-informative". As I continue learning about my data, do I update the prior?
As clicks and conversions come in, I have read that I have to "update" my distribution. Does this mean, that the parameters of my distribution changes, or that the prior changes? When I get a click for ad X, do I update more than one distribution? More than one prior?