Statistical significance of a sample

Question

Today, I was going through results of a survey when a colleague asked me, what is the statistical significance of the population cohorts in this survey.

I wanted to understand if anyone here could explain, what such a question as mentioned above would actually mean? I mean, it got me thinking and I was not able to reframe my colleague's answer in a way that would lead me to testing statistical significance.

So, let us say a population is made up of 4 segments namely - A,B,C,D. Now, if we take a sample out of this population with the 4 segments having certain sample sizes - how do we determine if these samples sizes are statistically significant. Does such a question even make sense to begin with.

Answer 1

What is the statistic that is of interest? What distribution do the data follow? One might want to quantify how significant. Etc.

Maybe this was clear in context (e.g. that you're trying to compare the means of some binomial distributions across four populations and your colleague trusts you to make some choices as to how significant). But without that, the question sounds a bit like "what is statistical hypothesis testing"?

Answer 2

To me, the question doesn't make sense because sample size is just one of the key drivers for your statistical test. To reject whatever null hypothesis you have, you may also need to consider:

• Effect size
• Design effects
• Sensitivity of your test
• Your significance level
• Your sampling scheme

How do you collect the samples? Are you doing random sampling? Are you doing stratification? Are you dividing the segments by clusters? If they're clusters, what about their intra-cluster correlation? Do you have any missing response? Do you want to weight your responses (very common in survey analytics, e.g. poststratification)?

I don't think you can say "I have collected X samples, and they are just significant."

Answer 3

One of the ways to interpret the question is to assume the colleague wanted to know if the sample size was 'significantly' large enough. It is a known and frequent question: how much data do we need to say something statistically meaningful? Some guidelines are here: http://conium.org/~maccoun/PP279_Lenth.pdf. If you want a number out of thin air, here is a funny answer: https://youtu.be/PbODigCZqL8. Seriously, I usually say a 100, for a guess.