Percent chance that two samples are different population (NPS)

Question

Practical question (not a statistician), I want to make a PPT slide (amongst several) in a business discussion, related to comparing a new survey with an old one. Want to be able to say something to the effect that increasing the size of my resample would not change the comparison (not a fluke in other words). Just some simple equation? Please?

First study, N=67, had 11 detractors on an NPS (net promoter score) metric. Second study, N=24 showed 0 detractors. [Which I knew it would as I had previous well-spread interviews, secondary reports, etc.]

The first study had a bunch of issues with its audience selection and questions (using very confusing terms, literally wrong ones, and likely mixing a bunch of beer drinkers into a whiskey survey and even making them think it was a general drinks survey--not really those products but analogous).

I've already panned the screen terminology and question terminology. And the free responses support my stance also. (Why do people try to get cute and act smart instead of foolproof rock simple? GRR!)

But I would like some simple equation I can run that says (everything the same, and I agree both population and method differ), what is the fluke random rolling dice chance, I'm wrong. In other words, my N=24 is insufficient. It has got to be freaking miniscule card-picking chance, like dealing a straight flush. I pretty much knew what would happen as I'd done N=8 in depth interviews in this area already and they were structured as VOC, but still had a lot of rating questions in there, just to spark discussion. And then I ran a disciplined, scripted, N=24 and got the same result (as my VOC, not the messed up Web quant survey).

Answer 1

It appears you identified $i=11$ detractors in a survey of $m=67$ people and later, in a separate survey of $n=24$ people, identified $j=0$ detractors; and you wonder how surprising that difference might be.

Perhaps the simplest way to conceive of (and therefore model) this situation is to suppose all $m+n=67+24$ people were available at the same time; that among them were $i+j=11+0$ detractors; and in a random sample of $m=67$ it happened, just by the luck of the draw, that a large number of $i$ or more detractors were picked up.

This chance is easy to compute from basic definitions, because the number of ways exactly $i$ detractors can appear in a sample of size $m$ is equal to (a) the number of ways all $i+j$ detractors can be split into a group of size $i$ (to be part of the first sample) and a complementary group of size $j$ (to be part of the second sample) times (b) the number of ways all $n+m-(i+j)$ other respondents can be split into a group of size $n-i$ (for the first sample) and a complementary group of size $m-j$ (for the second sample). Using the standard Binomial notation

$$\binom{x}{y}=\text{Number of subsets of size } y \text{ in a set of size }x = \frac{x!}{(x-y)!y!}$$

(as always, $x! = x(x-1)(x-2)\cdots(2)(1)$ is the factorial of $x$), we may write this count as

$$\binom{i+j}{i}\,\binom{(m+n)-(i+j)}{m-i}.$$

Because there are $\binom{m+n}{m}$ distinct ways to split all $m+n$ people into the two surveys, the probability is the proportion of all such ways comprised by the event in questions. It is computed as the ratio of the two counts,

$$p(i;m,n,j) = \frac{\binom{i+j}{i}\,\binom{(m+n)-(i+j)}{m-i}}{\binom{m+n}{n}}.$$

For the data in the question, $p(11;67,24,0)\approx 0.0272 = 2.72\%.$ Since it's impossible for any more than $11$ detractors to have been in the first sample, we don't have to add in any more possibilities: $2.72\%$ is the answer.

Although this chance is not as astronomically small as suggested in the question, it is small enough to suggest this probability model of the two samples is incorrect. Which part of it must fail? The assumption that chance alone was operating in placing all the detractors in the first sample. That justifies treating the difference in detractor counts as arising at least in part from some real difference in the populations, rather than attributing it to chance alone.