Every week, my $m-1$ friends and I enter a pub quiz which has $n$ points available. In any given week only some of us are there. Record the presence/absence of member $i$ in week $t$ in the matrix $x_{ti}$.
I'd like to build a predictive model for the number of points we score in any given week, depending on who's there. A simple model is to assume that each question is equally hard, and that player $i$ has probability $p_i$ of knowing any question. Then the probability that we get a given question correct is
$$q({\bf x},{\bf p}) = 1 - \prod_{i=1}^m(1-p_i)^{x_i}$$
Therefore, the probability that we score $k$ points is
$$P({\rm Score} = k|{\bf x},{\bf p}) = {n\choose k}q^k(1-q)^{n-k}$$
and the log-likelihood for a given ${\bf p}$ and ${\bf x}$ is
$$L = \log {n\choose k} + k\log q + (n-k)\log(1-q)$$
I can write a numerical routine which maximises this, to find the maximum-likelihood estimator for $\bf p$. However, the resulting estimate badly overfits the data (as can be detected by cross validation).
A good solution seems to be to introduce a penalty (regularization) term into $L$, which penalizes large or small probabilities. As I understand it, this is equivalent to having a prior on $\bf p$. However, I don't know what the form of this prior should be. Two simple choices for the penalty term are:
$$\| {\bf p} - 1/2\|^2$$
and
$$-\sum_i\log \left( \frac{p_i}{1-p_i}\right)$$
but these are very ad-hoc. I'd be interested to know what an appropriate conjugate prior is (assuming one even exists). Any hints?
