6
$\begingroup$

Consider a questionnaire where we ask someone about their sexuality. The five options, for simplicity, are:

  • Heterosexual
  • Homosexual
  • Bisexual
  • Other
  • 'Prefer not to say'

Assume we ask the population. We collect no other information about them except their sexuality.

We have reasonable suspicion that the 'prefer not to says' are not missing at random. We think that the probability of an individual selecting 'prefer not to say' will be higher for individuals who are homosexuals, bisexuals and other(s).

So if we strip out the 'prefer not to says' we will be reporting on a subset of the population which we know is skewed.

We would rather report on the data including the 'prefer not to says', incorporating our uncertainty of how they are distributed.

For example:

  • Heterosexual - 60%
  • Homosexual - 10%
  • Bisexual - 10%
  • Other - 10%
  • 'Prefer not to say' - 10%

In theory (though unlikely), every single 'prefer not to say' could be heterosexual. So we know that the percentage of heterosexuals in the population must lie between 60-70%.

However, can we do one better and report a confidence interval of some kind? All I could think of was creating a prior probability distribution for the 'prefer not to says' and creating a credible interval from that.

$\endgroup$
  • $\begingroup$ Possibly of interest: Dealing with 'don't know' answers for a categorical outcome variable. $\endgroup$ – gung - Reinstate Monica Oct 27 '14 at 21:17
  • $\begingroup$ i would prefer not to even take such questionares no matter how they are marketed :). Let me give you another example: If you draft for war in Iraq a) Go to front line, b) Go to back line c) Goto paramedics d) Prefer not to answer. Better not even take these questionaires $\endgroup$ – Nikos M. Oct 31 '14 at 7:11
  • $\begingroup$ i guess the 'other' option includes: 'bestiality', 'pansexuality', 'asexuality' (i'm sure we can squezze others things in there if needed) $\endgroup$ – Nikos M. Oct 31 '14 at 11:38
2
$\begingroup$

Answering your specific technical question -- the theory of inference for partially identified distributions has been developed in the works of Charles Manski. In your example, the confidence interval would be (60% minus 1.96 times whatever standard error corresponds to the 60% assumption, 70% plus 1.96 times whatever standard error corresponds to the 70% assumption).

You can do something different statistically, and treat "Prefer not to say" as missing data. Then you can impute the answers (better, using multiple imputation) based on additional covariates, such as behaviors and attitudes (towards religion, say), etc.

Finally, to rectify the problem at its source, you need to design your instrument differently. They say that there are no sensitive questions, but there are sensitive answers to some questions. The answer "heterosexual" is not particularly sensitive, but non-straight answers are. I am not a questionnaire designer, I can't guide you very far, but there are plenty resources from, you know, bound books as they used to have in the XX century, to MOOCs.

$\endgroup$
  • $\begingroup$ Tbanks for the response StasK. Could you summarise, briefly, why those confidence intervals apply? In my situation, I know the minimum percentage of heterosexuals must be 60% - won't a confidence interval of 60% minus 1.96*s.e be moving the wrong way? $\endgroup$ – NickB2014 Oct 31 '14 at 11:31
1
$\begingroup$

You can't get there from here, in my view. If there are very few prefer-not-to-say's, you can omit them without much bias. However, if you have lots in that category (say 5% or more), then I think you need a different design. Like ... don't even ask them if they "prefer not to say"!

There are ways of asking sensitive questions. In one method, people are told to answer truthfully or to lie, according as some random process (hidden to the interviewer) directs them. Using the known probability of a lie, you can infer the proportion with a particular characteristic without knowing the actual characteristics of any individual. This works for binary questions.

I like your idea of going Bayesian on this if you have prior information about the "prefer not to says".

$\endgroup$
  • 4
    $\begingroup$ Not supplying a "prefer not to say" can lead to outright lies for answers, which seems like it would make the data worse rather than better. And of course you can compute confidence intervals! The question is whether they would be narrow enough to be useful. $\endgroup$ – whuber Oct 27 '14 at 21:22
  • $\begingroup$ Is it a case therefore that I have to stick with my 100% confidence interval or make some distributional assumptions in order to use a confidence interval of <100% (in which case, using a 'prior' seems the best way). $\endgroup$ – NickB2014 Oct 31 '14 at 11:41
0
$\begingroup$

(add a statistical answer complementary to my political comment on top)

We have reasonable suspicion that the 'prefer not to says' are not missing at random. We think that the probability of an individual selecting 'prefer not to say' will be higher for individuals who are homosexuals, bisexuals and other(s)

Here one wants to reduce the 'prefer not to says' answers to other categories in a statisticaly correct and significant manner.

This would be correct under the following conditions:

1) The rest options are independent and elementary events of the sample space

2) The rest options partition the sample space completely (and representatively)

3) The prior probabilities of the rest options can be infered indepentantly

Then one can indeed reduce the 'prefer not to say' answers (or other types in same vain) to the elementary options (partitions) using the answer data to calibrate the reduction (within a statistical significant range)

For the example question given (and similar examples given in comments) this is not so. If one insists on that effectively endorses the truism that "there are lies, big lies and statistics".

From wikipedia Statistical Imputation

[M]ost statistical packages default to discarding any case that has a missing value, which may introduce bias or affect the representativeness of the results

i would prefer not to even take such questionares no matter how they are marketed. Let me give another similar example that makes the previous paragraph explicit:

If you draft for the war in Iraq:

a) Go to front line

b) Go to back line

c) Goto paramedics

d) Prefer not to answer.

Better not even take these questionaires!

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.