Each member of a small group composed of 10 men and 10 women answered a long questionnaire. The questionnaire is made of numerous (e.g., 1000) two-choice questions about preferences such as ‘blue or green?’, ‘Seinfeld or Friends?’ and so on.

Given the responses of a new participant to the same questionnaire, we’d like to estimate the probability that this participant is a woman.

How should this be done?

Below is my novice attempt at applying a Bayesian model selection approach and where it fails (tl;dr - inter-item correlations are hard to account for).

Bayes tells us that:


So the critical ingredient are the likelihood probability functions $p(Responses|Woman)$ and $p(Responses|Man)$).

We'll try to build this up from the likelihood of a particular response - e.g., $p(Seinfeld|Woman)$.

Since we had only 20 participants, we can’t safely use the ratio of Seinfeld responses among the 10 women as a Bernoulli success probability $p_{Seinfeld;Woman}$ (the probability of responding Seinfeld given the participant is a woman) - due to the small sample size, there's uncertainty about this ratio in the population. However, if we assume a uniform prior (beta distribution with $\alpha=1$ and $\beta=1$) over the $p_{Seinfeld;Woman}$, then the Posterior Predictive Distribution is Bernoulli distributed with $p_{Seinfeld;Woman}=\frac{n_{Seinfeld;Woman}+1}{n_{Seinfeld;Woman}+n_{Friends;Woman}+2}$ (the $n$'s are the counts of 'Seinfeld' and 'Friends' responses among our 10 women).

So we have an analytic solution for $p(Seinfeld|Woman)$. But how do we get from that to $p(Responses|Woman)$?

Simply taking the product of the individual responses unwarrantably assumes independence between items. In other words, $p(Responses|Woman)=\prod_\limits{i}p(Response_{i}|Woman)$ is not necessarily correct.

To illustrate this issue, let's suppose that the responses to the question ‘Breaking Bad or The Sopranos’ are perfectly correlated with the responses to ‘Seinfeld or Friends’. Then, collecting the responses to the two questions won't increase the evidence compared with collecting a single response, yet the multiplication of individual item probability erroneously assumes it does. This might lead to over-confidence of our model's decision.

So how inter-item dependencies should be taken into account? Explicitly modeling the 1000x1000 correlation matrix seems somewhat impractical. And even if it’s feasible for this binary case, it doesn’t seem to be scalable to the more general multinomial case (consider questions like 'Seinfeld, Friends, Cheers or Frasier?').

The only solution that comes to my mind is calibrating the probability estimate resulting from the product using a logistic regression fitted in some cross-validation scheme. Any better ideas? Alternative approaches? Can a non-parametric model save the day?


The basic problem you encounter here is that you have a low sample size, and yet you want to take account of a large number of possible interactions; this is not a realistic expectation. Even when you are using Bayesian methodology, there is a negative consequence to using over-identified models. Parameter estimation is possible in such cases, but the estimates are largely determined by the priors, and so they are not robust. There is some literature on over-parameterisation in a Bayesian context, and this might be of assistance (see e.g., reference below).

If you are dealing with $k$ binary variables and you want to measure all interactions between them, you will have $2^k - 1$ coefficients representing main effects and all interaction effects (this leaves out the base term). The number of model terms at different levels of interaction is given by the binomial coefficients:

$$\begin{matrix} \text{TERMS} & & & \text{NUMBER OF TERMS} \\ \text{Base term} & & & {k \choose 0} = 1 \\ \text{Main effects} & & & {k \choose 1} = k \\ \text{Interaction of 2 variables} & & & {k \choose 2} = k(k-1)/2 \\ \text{Interaction of 3 variables} & & & {k \choose 3} = k(k-1)(k-2)/6 \\ ... \\ \text{Interaction of } k-1 \text{ variables} & & & {k \choose k-1} = k \\ \text{Interaction of } k \text{ variables} & & & {k \choose k} = 1 \\ \text{TOTAL} & & & 2^k \text{ terms} \\ \end{matrix}$$

As you can see, adding higher levels of interaction rapidly increases the number of terms to estimate. Even if you only went up to an interaction of pairs of variables, that already gives you a large number of model terms. For $k = 1000$ questions, taking main effects plus paired interactions gives $499500$ model terms. Attempting to incorporate all these paired interactions in your model would already lead to a case where you lack identifiability. This is just with paired interactions - if you go to higher-order interactions it gets much worse!

In view of this issue, I do not think a model with interactions between the questions is going to be useful for the small amount of data you have. With such a small amount of data you would run into identifiability problems, and these would mean that your inferences would be highly sensitive to prior specification. I would suggest that falling back on an assumption of independence between the questions would be reasonable; it is unlikely that this independence assumption actually holds, but with the small amount of data you have, there is no way for you to make a robust estimate of main effects and interactions.

[1] Gustafson, P. (2009) What are the limits of posterior distributions arising from non-identified models, and why should we care? Journal of the American Statistical Association 104(488), pp. 1682-1695.

  • $\begingroup$ Thank you for the detailed answer. I agree that the model identifiability problem here cannot be solved for the given sample size. Yet - this is not the task at hand - I don't really need a good estimate of P(Responses|Woman) - I'm looking for a good estimate of P(Woman|Responses). The naive Bayes solution (assuming independence) simply won't work here since it's easily biased by inter-item correlations. Instead, if I trained a (strongly regularized) logistic regression on these data, it would be more robust to inter-item correlations. However, I'm wondering if there's a smarter approach. $\endgroup$ – Trisoloriansunscreen Mar 5 '18 at 4:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.