2
$\begingroup$

From a real-life situation, but distorted enough to be untraceable...

A cohort of patients (n= 300) were each one subjected to two procedures (A and B) substantially separate in time not to interfere (let's just assume). From the patients' perspectives the procedures were identical. However, there was a technical difference in the utilization of different medications (same class of drugs, but different) during the performance of the procedures to suspect that the results could differ.

The patients were contacted after both procedures had already taken place, and asked what their recollections were regarding their response to procedure A and B. And the options were (simplifying a bit): A much better than B; A better than B; A same as B; B better than A; and B much better than A. Seemingly the results were pooled, obtaining that 65% favored A; 25% favored B; and 10% felt similar effects. It was concluded that there was a statistically significant difference favoring A.

I have read the post on a similar situation, resolved applying a $binomial$ with an assumed $p$ of success of $0.5$, and only considering responders with a preference. However, in that case (preference of chicken v beef) the "undecided" could safely be left out. This doesn't seem right in our example.

What would be the test or tests appropriate to work out this problem? It can be either $H_0: A = B$ and $H_1 A \neq B$, or $H_1: A > B$.

It is likely that the the chicken v beef example is not essentially different from this case in terms of statistical approach. However, I can't but sense a difference in the information available before the experiment that would fail to be mathematically encapsulated if the "no difference" group was simply eschewed: in the case of the chicken v beef, we know that there is an intrinsic difference that may or may not be translated into preferences, whereas in the case of the two procedures A and B there was supposed to be no difference. The burden of proof should be higher in the second scenario. For instance, patients with chronic conditions tend to feel worse over time, and repeat procedures may yield decreasing returns; or there may be biases introduced due to cognitive expectations. In particular, in the case presented all patients had procedure A before B.

The numbers I provided seem to indicate that A is better, and that discarding the 10% of indifferent responders wouldn't make a difference, but what if there were 50% of no-difference answers, 45% favoring A and 5% favoring B? Wouldn't any conclusion of A > B be more than suspect, even though we would have 135 patients favoring A with a probability of getting this result or higher with a p of success of 0.5 of 

pbinom(135, 150, 0.5, lower.tail=F)
[1] 1.396379e-26

?

Improve this question
$\endgroup$
11
  • $\begingroup$ "beef" rather than "soup" $\endgroup$
    – Glen_b
    Commented Aug 15, 2015 at 4:36
  • $\begingroup$ Can you elucidate why you don't feel that leaving out people who don't express a preference is inappropriate? Of course it affects both the specific hypothesis being tested ("Among those expressing a preference ...") and the conclusion, but you haven't clearly explained what your true comparison is in a way that clearly identifies anything else should be done. Note that if you consider the particular contrast $\pi_A-\pi_B$ and perform your test on that, you get exactly the same outcome as simply saying "Among those who express a preference". $\endgroup$
    – Glen_b
    Commented Aug 15, 2015 at 5:24
  • 1
    $\begingroup$ I think there is a substantive question here, just different enough so that my answer to the earlier question doesn't fully address it, which is when and how to worry about the category which has no preference. I think an answer would likely tend to focus on how thinking about "what sort of statement do you want to make?"/"what precise question are you answering?" relates to how you treat that category, but in the context of the linked question would need to treat the other kinds of questions one might ask/statements one might make. The problem with me editing your question to make your ...ctd $\endgroup$
    – Glen_b
    Commented Aug 16, 2015 at 2:35
  • 1
    $\begingroup$ I would probably tell the textbook authors they're probably answering a question they don't actually care about the answer to; the answer isn't in coming up with new ways to play with the data, but to ask the right question of the data you have. Which is to say, to be more precise about what we want to find out. Assuming some people don't have a preference, what is it we actually want to know? Is it whether more people prefer A to B than prefer B to A? Is it "Do a majority prefer A?" (a quite different question!) or something else? $\endgroup$
    – Glen_b
    Commented Aug 16, 2015 at 3:57
  • 1
    $\begingroup$ I'm not requesting clarification of any statistical issue, but a subject matter issue (about which you know and I do not) plus basic definitional logic relating to what conditions are included in a question relating to categories (which may include your interests alongside subject matter considerations). What's the actual material question in relation to the subject area (not the statistics) that you want to answer? e.g. what's the actual research hypothesis, carefully stated (in plain words, while hopefully avoiding ambiguities about what category things are in)? $\endgroup$
    – Glen_b
    Commented Aug 16, 2015 at 4:45

1 Answer 1

Reset to default
4
$\begingroup$

I'm still pondering some additions to this; it's not quite adequate yet.

Ultimately such questions come down to "what do you want to know"?

  1. If the question is purely about comparing preferences between A and B in the presence of people that genuinely don't have a preference, those people don't enter into the question of interest. That is, we have 3 categories (A is preferred, neither is preferred, B is preferred), and population proportions $\pi_A,\pi_\varnothing,\pi_B$ but the question of interest relates to whether the contrast $\pi_A-\pi_B$ is different from $0$ ... a question that says nothing whatever about the third category. That question on that contrast of proportions of preferences can be tested quite readily.

    This would, for example, be of interest if you wanted to identify if there was a preferred treatment in order to displease the fewest people (presumably the proportion choosing A will be happier if you choose A and the proportion not expressing a preference would be equally happy either way, so they don't contribute to the question of minimizing displeasure).

  2. If the question is about whether a majority of people prefer A (say), that's quite a different question, one that would be of the form $H_0: \pi_A\leq \frac12$ vs $H_1: \pi_a>\frac12$.

    This would tell you whether more than half would be actively pleased to get treatment A rather than the alternative. It's not clear to me that this is really a question of subject matter interest, but perhaps there's some circumstance where it is (this might be where the question's mention of 'burden of proof' comes in ... but to be honest that really sounds like it's hinting at some kind of equivalence test).

  3. If the question is about whether a majority of people prefer A or don't care, that's a similar question to 2., one that would be of the form $H_0: \pi_A+\pi_{\varnothing}\leq \frac12$ vs $H_1: \pi_a+\pi_{\varnothing}>\frac12$.

    This would be something like "Do we at least avoid displeasing more than half?". However, this turns out to be entirely equivalent to testing a hypothesis like 1.

  4. To start playing with the data (as mentioned in comments) tends to suggest that the experimenters don't believe the patients when they say they don't have a preference. They in effect insist that the patients must have a preference and attempt to assign them one in an arbitrary way. I regard this as both logically and ethically somewhat questionable. It's not clear how this really corresponds to a subject-matter question anyone would really care to know the answer to*, but maybe someone can construct one that doesn't seem quite as odd as this currently strikes me.

* "When a proportion of the population are arbitrarily assigned a preference in spite of saying they didn't prefer one, more than half preferred A" is not a question I at least would care to know the answer to. What practical problem is addressed by it?

At first glance it seems rather similar to the old joke about strawberries and cream

“Comes the revolution,” the orator declared, everyone would live the good life and eat strawberries and cream. “I don’t like strawberries and cream!” responded one of his listeners. “Comes the revolution,” the orator declared, “You’ll eat strawberries and cream—and like it!”

(just in case the analogy is unclear; the equivalence is from having preferences to liking strawberries and cream)

--

Order effects

In the preceding discussion, I ignore the issue of the order effects.

Order effects certainly will matter in a way that no amount of fiddling with the no-preference group could remedy.

If the A treatment is always first, then the treatment difference is confounded with any order effect, and we literally have no way to disentangle the two with that data.

If even a few people had the treatments in the other order, something might be done, but without that you're left with at best non-statistical arguments (such as subject-matter arguments) as to what the order effect might be, if any.

Improve this answer
$\endgroup$
5
  • $\begingroup$ What a great answer! Just on point 4, and to be clear, the point the authors want to make is that A > B. My role is to make sure the claim is substantiated by the data. I take from your parsed analysis that the "don't care/don't recall" group can be safely put aside, and don't cast doubt on any possible "order" effects... "I used to feel so much better..." type of reasoning... due to the fact that A always came before B. $\endgroup$
    – Antoni Parellada
    Commented Aug 16, 2015 at 12:20
  • $\begingroup$ You are making me wonder now about the idea of a density and a distribution... in the prior comments... I guess a distribution does have to add up to 1. Thanks! $\endgroup$
    – Antoni Parellada
    Commented Aug 16, 2015 at 12:23
  • $\begingroup$ I didn't address the issue of order effects at all. I pretended they didn't exist for the purpose of the discussion. I should have come back and discussed that. I will add a little now, and perhaps more later. $\endgroup$
    – Glen_b
    Commented Aug 16, 2015 at 12:38
  • $\begingroup$ I can't but see the people on the fence trying to tell us that whatever the relative score of A v B turns out to be its significance has to be brought down a notch. It's like winning an election in a landslide with a very low participation rate. Is the winner so much better, or there's some technical explanation difficult to point out to account for his victory, even though all the candidates are equally poor and without substantial differences from the point of view of the population at large? $\endgroup$
    – Antoni Parellada
    Commented Aug 16, 2015 at 13:30
  • 2
    $\begingroup$ It would perhaps be more like knowing that your low-turnout results came almost entirely from a single state .... but without being able to tell which state. $\endgroup$
    – Glen_b
    Commented Aug 16, 2015 at 14:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.