From a real-life situation, but distorted enough to be untraceable...
A cohort of patients (
n= 300) were each one subjected to two procedures (
B) substantially separate in time not to interfere (let's just assume). From the patients' perspectives the procedures were identical. However, there was a technical difference in the utilization of different medications (same class of drugs, but different) during the performance of the procedures to suspect that the results could differ.
The patients were contacted after both procedures had already taken place, and asked what their recollections were regarding their response to procedure
B. And the options were (simplifying a bit):
A much better than B;
A better than B;
A same as B;
B better than A; and
B much better than A. Seemingly the results were pooled, obtaining that
65% favored A;
25% favored B; and
10% felt similar effects. It was concluded that there was a statistically significant difference favoring
I have read the post on a similar situation, resolved applying a $binomial$ with an assumed $p$ of success of $0.5$, and only considering responders with a preference. However, in that case (preference of chicken v beef) the "undecided" could safely be left out. This doesn't seem right in our example.
What would be the test or tests appropriate to work out this problem? It can be either $H_0: A = B$ and $H_1 A \neq B$, or $H_1: A > B$.
It is likely that the the chicken v beef example is not essentially different from this case in terms of statistical approach. However, I can't but sense a difference in the information available before the experiment that would fail to be mathematically encapsulated if the "no difference" group was simply eschewed: in the case of the chicken v beef, we know that there is an intrinsic difference that may or may not be translated into preferences, whereas in the case of the two procedures
B there was supposed to be no difference. The burden of proof should be higher in the second scenario. For instance, patients with chronic conditions tend to feel worse over time, and repeat procedures may yield decreasing returns; or there may be biases introduced due to cognitive expectations. In particular, in the case presented all patients had procedure
The numbers I provided seem to indicate that
A is better, and that discarding the
10% of indifferent responders wouldn't make a difference, but what if there were
50% of no-difference answers,
B? Wouldn't any conclusion of
A > B be more than suspect, even though we would have
135 patients favoring
A with a probability of getting this result or higher with a
p of success of
pbinom(135, 150, 0.5, lower.tail=F)  1.396379e-26