Practical question (not a statistician), I want to make a PPT slide (amongst several) in a business discussion, related to comparing a new survey with an old one. Want to be able to say something to the effect that increasing the size of my resample would not change the comparison (not a fluke in other words). Just some simple equation? Please?
First study, N=67, had 11 detractors on an NPS (net promoter score) metric. Second study, N=24 showed 0 detractors. [Which I knew it would as I had previous well-spread interviews, secondary reports, etc.]
The first study had a bunch of issues with its audience selection and questions (using very confusing terms, literally wrong ones, and likely mixing a bunch of beer drinkers into a whiskey survey and even making them think it was a general drinks survey--not really those products but analogous).
I've already panned the screen terminology and question terminology. And the free responses support my stance also. (Why do people try to get cute and act smart instead of foolproof rock simple? GRR!)
But I would like some simple equation I can run that says (everything the same, and I agree both population and method differ), what is the fluke random rolling dice chance, I'm wrong. In other words, my N=24 is insufficient. It has got to be freaking miniscule card-picking chance, like dealing a straight flush. I pretty much knew what would happen as I'd done N=8 in depth interviews in this area already and they were structured as VOC, but still had a lot of rating questions in there, just to spark discussion. And then I ran a disciplined, scripted, N=24 and got the same result (as my VOC, not the messed up Web quant survey).