6
$\begingroup$

Every week, my $m-1$ friends and I enter a pub quiz which has $n$ points available. In any given week only some of us are there. Record the presence/absence of member $i$ in week $t$ in the matrix $x_{ti}$.

I'd like to build a predictive model for the number of points we score in any given week, depending on who's there. A simple model is to assume that each question is equally hard, and that player $i$ has probability $p_i$ of knowing any question. Then the probability that we get a given question correct is

$$q({\bf x},{\bf p}) = 1 - \prod_{i=1}^m(1-p_i)^{x_i}$$

Therefore, the probability that we score $k$ points is

$$P({\rm Score} = k|{\bf x},{\bf p}) = {n\choose k}q^k(1-q)^{n-k}$$

and the log-likelihood for a given ${\bf p}$ and ${\bf x}$ is

$$L = \log {n\choose k} + k\log q + (n-k)\log(1-q)$$

I can write a numerical routine which maximises this, to find the maximum-likelihood estimator for $\bf p$. However, the resulting estimate badly overfits the data (as can be detected by cross validation).

A good solution seems to be to introduce a penalty (regularization) term into $L$, which penalizes large or small probabilities. As I understand it, this is equivalent to having a prior on $\bf p$. However, I don't know what the form of this prior should be. Two simple choices for the penalty term are:

$$\| {\bf p} - 1/2\|^2$$

and

$$-\sum_i\log \left( \frac{p_i}{1-p_i}\right)$$

but these are very ad-hoc. I'd be interested to know what an appropriate conjugate prior is (assuming one even exists). Any hints?

$\endgroup$

1 Answer 1

5
$\begingroup$

There's no conjugate prior for this likelihood. Likelihoods that admit conjugate distributions correspond to data distributions that are members of some exponential family. Having a non-linear function of the parameters in the log-likelihood makes it impossible for the data distribution to belong to an exponential family.

Even though there's no conjugate prior, one possibility for a reasonable log-prior is

$L_0({\bf p} ; {\bf j}, {\bf N} ) = \sum_i [j_i \log(p_i) + (N_i - j_i) \log (1 - p_i)]$

You can think of this log-prior as equivalent to a log-likelihood for a data set in which each person did a set of questions alone and correctly answered $j_i$ out of $N_i$. This interpretation allows you to set the prior parameters ${\bf j}$ and ${\bf N}$ in a reasonably intuitive way. I'd be somewhat surprised if even small values of $N_i$ (e.g., 2 to 4) did not provide good regularization. Note that $j_i$ and $N_i$ need not be integers.

It seems to me that you're thinking of using the plug-in predictive distribution. May I suggest you go full Bayes and use the posterior predictive distribution instead? It would require MCMC, which may be more trouble than you're willing to go to. (If you're using Matlab I can recommend an MCMC routine that would shorten your coding time considerably.)

$\endgroup$
3
  • 2
    $\begingroup$ I think your affirmation "Likelihoods that admit conjugate distributions correspond to data distributions that are members of some exponential family" is not entirely correct. This family has a natural conjugate prior but the definition of conjugate prior is not limited to the exponential family, though it is difficult/impractical for most distributions out of this family. (+1) for the rest of the answer. $\endgroup$
    – user10525
    Aug 13, 2012 at 10:39
  • 1
    $\begingroup$ @Procrastinator, can you give me some examples? I don't doubt you -- I just want to increase my knowledge. $\endgroup$
    – Cyan
    Aug 13, 2012 at 13:13
  • 2
    $\begingroup$ I do not remember any reference of this at the moment but you can find a discussion on this here or in the reference of the creators of the concept here. $\endgroup$
    – user10525
    Aug 13, 2012 at 13:40

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.