I have a dataset with several observed view rates per unit for products. I want to use this rates as proxies for the demand for a certain product.

Product  Items  Views  Rate(Views/Item)
a           50    800    16
b            2     80    40
c          100   1200    12

However, for example, in the case of product "b" only few data was observed. Hence, the observed rate is very unreliable compared to other products that have much more observations.

Therefore, I specifically need a procedure to adjust the rates for products with few items to account for the fact, that the observed rates are not very certain, but are probably more like some average rate.

My idea: using a prior rate (or distribution to be more concise) that stems from all products (i.e., average rate of 13.68 in this case), I want to calculate a posterior rate for every single product using bayesian statistics.

Nonetheless, because the rates do not lie within [0..1] as for binomial distributions (and also a binomial distribution does not fit in this case), updating by estimating Alpha/Beta-parameters of a Beta-Distribution and calculating posterior rates by (alpha + x) / (alpha + beta + n) does not work.

What is a suitable way to tackle this issue? What prior distribution to use and what method to use to obtain posterior rates for each product that adjust the cases with few observations to be closer to an average?


It depends a bit what makes sense here, but if you think that what you are interested in is views per item, then a Poisson distribution may make sense. I.e. $$\text{Views} \sim \text{Poisson}(\text{items} \times \lambda).$$ For a single Poisson rate, $\lambda$ - or in the case that you think that the views/item rates have nothing to do with each other for the different products - a Gamma distribution would be conjugate. If you parameterize it in terms of shape $\alpha$ and rate $\beta$, then the posterior has shape parameter $\alpha+\text{Views}$ and rate parameter $\beta+\text{items}$.

If you think that the rates might be similar (in the sense of exachangeability that you could not predict up-front which rate would be lower or higher than the others), you might want to go down the empirical Bayes route. In that case you would estimate the mean rate $\mu$ and dispersion parameter $\kappa>0$ (or $\theta:=1/\kappa>0$ if you e.g. use glm.nb in R) considering the views as observations from a negative binomial model with an intercept and a $\log(\text{items}_i)$-offset. Then you would use independent $\text{Gamma}(\alpha=\hat{\theta}, \beta=\hat{\theta}/\hat{\mu})$ priors for each individual rate and would update them individually.

Or you could formulate this explicitly as a hierarchical Bayesian model. It should be relatively easy to fit, because a $\lambda_i \sim \Gamma(\alpha, \beta)$ and $\text{views}_i \sim \text{Poisson}(\lambda_i \text{items}_i)$ allows Gibbs sampling (so you can use any Gibbs sampler, or of course any general purpose MCMC sampler such as Stan).


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.