# Hypothesis testing with a response scale of "1, 2, 3, 4, or not applicable"

Let's imagine I'm testing a new drug that purports to reduce the effect of a hangover when the drug is administered shortly before an episode of heavy drinking.

On Friday night, I stake out the entrance to a frat party and, using random assignment, give each new guest either a placebo or the active drug. The next morning, I ask each participant to complete this two-item questionnaire:

1. Did you get drunk last night?

• no
• yes
2. If "yes," how hungover do you feel this morning?

• 1: not at all
• 2
• 3
• 4: extremely

Normally I'd perform a simple t-test for the difference in mean response to the Likert scale.

But in this case, the drug could have some effect in discouraging people from getting drunk in the first place. Anyone who answers "no" to Question 1 necessarily has missing data for Question 2.

How can I do the usual hypothesis test -- that the drug does/does not differ from placebo -- given these idiosyncrasies?

• Regardless of your context, there is nothing to compare here. Whoever designed the questionnaire built a dead end. The only possibility is that some people said No to Q1 but answered Q2 any way. If you are an optimist, they provide data for a comparison. If you are a pessimist, nothing they said is credible as they evidently didn't understand the instructions. Commented Sep 2, 2013 at 22:44

I agree with the others that the design may not be optimal. But if it happens that you have such kind of data, what about a Heckman selection model. This is a well known model in econometrics (Nobel Prize) to account for the fact that some kind of selection process leads to a non-random sample introducing bias. It is a two-stage method: First, it is modeled whether someone got selected in the final sample (e.g., drunk vs. not drunk), which is in most cases accounted for by a probit model (dichotomous DV). Second (but simultaneously), the actual DV of interest (e.g., hangover) is modeled using a second regression equation with the two error terms being correlated.
Not being a statistical expert in Heckman models, I can only refer to the original paper, wiki, this site, ... I think it is implemented at least in R and Stata.
Heckman, J. (1979). Sample selection bias as a specification error. Econometrica, 47, 153–61. doi:10.2307/1912352

From a statistical standpoint, there is no problem only using data points gathered from people who got drunk. In other words, discard those responses who did not. The hypothesis test assumptions still hold. Use a t-test, ANOVA, whatever your heart fancies.

From an experimental standpoint, you have no control group, which is a bad idea.

From a social science standpoint, you have changed people's behavior by asking them to participate before they begin drinking, which means your frequencies will not match reality.

Instead, you could ask people as they exit a bar, or exit a party. But, you'd have to deal with drunken people who may not be in any condition to make promises or evaluations.

How would the drug discourage people from drinking? Wouldn't drinking less be an effective way to avoid hangover and, as such, a useful effect of the drug? Could the participants know/guess to which group they were assigned? In any case, I don't see any major difference with a regular clinical study.

Importantly, “Did you get drunk?” seems just as subjective as “Do you feel X?” Also, I don't know how the questions were presented but I don't see any a priori reason why you would necessarily have missing data on the second question. Even people who did not drink any alcohol, let alone “get drunk”, can meaningfully report that they are not feeling hungover. You could therefore look at it as a study with two equally subjective outcomes (“remembering getting drunk” and “feeling something like the effect of drunkenness”).

If you want to compare the subjective experience of people who were exposed to the same amount of alcohol, you would have to ask about what they actually drunk or, better yet, observe/measure/manipulate that yourself as recollections seem especially problematic in this context.

Incidentally, asking about specific symptoms (headache, etc.) would seem preferable as partying/lack of sleep could also have unpleasant effects and the question as currently formulated invites speculation on the part of the participants regarding the origin of these effects. This would also make it easier to ask the same questions to participants would did and did not drink alcohol and use the latter as a meaningful comparison group to discount pseudo-hangover effects.

In any case, you can certainly compare placebo and treatment groups on these two variables. If treatment assignment was not blind, you have some extra interpretation problems but no additional statistical difficulty.