Consider a questionnaire where we ask someone about their sexuality. The five options, for simplicity, are:
- 'Prefer not to say'
Assume we ask the population. We collect no other information about them except their sexuality.
We have reasonable suspicion that the 'prefer not to says' are not missing at random. We think that the probability of an individual selecting 'prefer not to say' will be higher for individuals who are homosexuals, bisexuals and other(s).
So if we strip out the 'prefer not to says' we will be reporting on a subset of the population which we know is skewed.
We would rather report on the data including the 'prefer not to says', incorporating our uncertainty of how they are distributed.
- Heterosexual - 60%
- Homosexual - 10%
- Bisexual - 10%
- Other - 10%
- 'Prefer not to say' - 10%
In theory (though unlikely), every single 'prefer not to say' could be heterosexual. So we know that the percentage of heterosexuals in the population must lie between 60-70%.
However, can we do one better and report a confidence interval of some kind? All I could think of was creating a prior probability distribution for the 'prefer not to says' and creating a credible interval from that.