# Tag Info

51

I agree with many of the other answers here but think the statement is even worse than they make it out to be. The statement is an explicit version of an implicit claim in many shoddy analyses of small datasets. These hint that because they have found a significant result in a small sample, their claimed result must be real and important because it is '...

35

"It's counterintuitive, but the smaller the sample size of a clinical test, the more significant its results are. The differences in a sample of 20 people may be more significant than in a sample of 10,000 people. If we need such a sample, there is a risk of being wrong. With 10,000 people, when the differences are small, sometimes they don't exist." I ...

24

(I think the phrasing is deliberately provocative.) If you have 10 observations and want to show that their mean is not zero, it will have to be quite a bit different from 0 if you want to have any kind of chance (power) of detecting the difference. If you have a trillion observations and want to show that their mean is not 0, the mean could be just a tiny ...

14

No. Many data sets can yield the same mean, SD and n. In the graph above, the three data sets on the left of each graph all share the same mean, SD and n. So do the three data sets on the right of each graph. This is Figure 1 from: Weissgerber, T.L., Milic, N.M., Winham, S.J., and Garovic, V.D. (2015). Beyond bar and line graphs: time for a new data ...

14

Given two samples that have the same mean, standard deviation, and N: are the values in each sample identical? In general, not unless N=2 in both samples. If N is larger than 2, they can differ. You can see this simply by trying it with some simple cases. Perhaps the easiest case is to take an asymmetric sample of size $N=3$ and flip it around its mean ($2,... 11 Can you confirm that it is a FALSE statement in statistics I think the statement is phrased poorly. In this context, the word "significant" seems to have the flavor of "importance". Difference detected in smaller datasets are not somehow more important or meaningful by virtue of being detected in small datasets. Rather, differences detected in small ... 10 Standard deviation does not decrease with sample size. The bigger your sample is, the closer the standard deviation should be to the standard deviation of the population. What follows, with larger sample size the spread of the standard deviations estimated on larger vs smaller samples would decrease, because based on larger samples we would get more precise. ... 8 The quote in question seems to originate from marianne.net (in French) and, as it stands, is definitely wrong. But, as Demetri and Dave pointed out, with some language bending there might be some truth to it. In my understanding, Prof. Raoult confuses significance and effect size. In a small sample, the effect size has to be large (i.e. of practical ... 8 Standard error decreases as the sample size increases. Standard deviation is a related concept but perhaps not related enough to warrant such similar terminology that confuses everyone who is starting to learn statistics. A sampling distribution is the distribution of values you would get if you repeatedly sampled from a population and calculated some ... 6 A typical way to use the sample size from the normal assumption would be to approximate the number of degrees of freedom in the t-distribution. Say your sample size calculation, using the normal distribution, gives a required sample size of 60. You would then use$n=60$and$df=59$in the sample size calculation with the$t_{df}$distribution. Yes, you ... 6 Although you are conducting novel research you should still be able to come up with plausible range of what the parameter values should be. So, based on these ranges, you would then conduct a number of simulations, where you systematically use values from within these ranges. You will then arrive at results where each combination of parameter values gives a ... 6 Nice experiment. The blue lines will be at$\mu \pm z_{\alpha/2} \sigma/\sqrt{n}$where$\alpha = 0.05$and$\alpha \mapsto z_\alpha$is the upper quantile function of the standard normal and$n$is sample_size_. In this case, this is exact since you are generating from the normal distribution and the sample mean will have the distribution$N(\mu, \sigma^2/...

5

All sample size calculations are built on top of a proposed inference that will be made from the data. This might be a confidence interval for an unknown parameter, or a hypothesis test for a set of hypotheses, or a Bayesian posterior inference, etc. Whatever the inference being made, there will be some appropriate measure of how "accurate" the inference ...

5

A smaller sample size is not better. A small sample size needs a more significant* result if you want to draw a conclusion from it. Let's consider some results and their interpretation: If your drug cures 30% of 10 people, the percentage of the general population cured could be anywhere between around 0% and 65% of people. If your drug cures 30% of 10000 ...

5

It's your power. Note in that first calculator link you can set the power downward to 60%, and that lowers the sample needed to 3310. If you could lower the power there to 50%, you would likely get 2725. When you make that significance test using the second calculator, you are at a critical point. If you had one less positive result, you wouldn't be ...

5

The way you describe it, i.e. estimating the effect size based on the data and then checking post-hoc the power based on the estimated effect size is how post-hoc tests are often done. Unfortunately, it is NOT how post-hoc power estimates should be done. See for example O'Keefe (2010). The reason for this is, that a post-hoc power analysis of this type ...

5

As in all research, you should specify your test and your level of statistical significance prior to analyzing your data. So if you used $\alpha = 0.01$ to figure out sample size calculations, then that's presumably the level you choose for your study, and so you should compare it to that. Otherwise why was that the $\alpha$ you choose for sample size ...

5

There is no simple answer to such question. You can train neural network with one sample, you'd just overfit to it. Moreover, there are some recent results that in some cases neural networks with few orders of magnitude more parameters than samples can achieve better test set performance than smaller networks. Such rules of thumb don’t even work for much ...

5

Some care is needed when talking about samples size in the context of mixed models. First, there is the overall (total) sample size, let's call it $N$ Then there is the number of subjects (cities in the case of your example), let's call it $n$ Then there is the number of observations within each subject (city). In observational studies this will often be ...

5

I would just explain what happened. You powered for N, and you got N*. It's not the first time this has happened (and won't be the last). Post hoc power would not be especially useful (as you have realized.)

4

There are a few things that are true, and worth understanding for where the confusion might slip in. First, it is possible to get high levels of confidence from small samples, so long as the effect is sufficiently pronounced. For example, a treatment that goes from 10% control recovery to 90% experimental recovery will show up with a very good score even if ...

4

The best way to do a sample size / power calculation for mixed models is via simulation. This entails the following steps: You simulate from the given design of interest. This will specify among others the number of groups/clusters/subjects and the number of measurements per cluster. You fit the model and perform the hypothesis testing of interesting and ...

4

You specifically ask about simulation. Following @Dave's Answer (+1), here are a couple of simulations in R. Suppose I take a million samples of size $n = 16$ from a population distributed as $\mathsf{Gamma}(\mathrm{shape} = 4,\, \mathrm{rate}=.1),$ so that the population mean is $\mu = 40$ the population variance is $\sigma^2 = 400,$ and $\sigma = 20.$ Then ...

4

Mixed models are good at coping with unbalanced designs. This is one of their advantages compared to other approaches such as ANOVA-type models. So I would not worry about this. You mention small group sizes, but the numbers you mention, are not, in my opinion, small. You will see in my answer to your other question that I was simulating data with maximum ...

4

Elaborating a bit on Jeremy's answer, let's think for a minute about what a power analysis is. The purpose is to determine how many participants one would need to "detect" an effect of a specific size. So in discussing the results of your experiment vis a vis the sample size you originally designed, and what the pandemic (unforseen circumstances) ...

4

What is the minimum sample size depends on the question: "Minimum sample size to accomplish what?". A paired-t test can be done on as few as 2 pairs if the only goal is to be able do some computations and get an answer (and you do not care about the quality of the answer). If the question is how big a sample size do you need for the Central Limit ...

3

Peter cannot inflate his effective sample size for the purpose of estimating treatment effects just by repeatedly subsampling the same experimental units -- this would be the most egregious form of 'pseudoreplication'. Sample size in the context of a designed experiment is set by the randomization design -- since the different samples within the same plot ...

3

This statement is misleading because it is unclear what he means by significant. In the case of a clinical trial, what you want to show is that people are more likely to heal when given a test treatment than when given a placebo. So you have two (random) groups of equal size, one of which gets a the treatment while the other gets a placebo. Then you observe ...

3

Before really answering the question, I have to point out that the study being discussed was a non-randomized open label study where the controls were possibly in a different facility than the treatment subjects, and they tossed out dead, ICU patients and dropouts due to side effects from the treatment group. The poor outcomes (it appears) came from the ...

3

A) What do you mean by "this situation?" If the issue is calculating sample size for a non-inferiority trial with a time-to-event outcome, I suggest chapter 7 of "Chow SC, Wang H and Shao J, 2007. Sample size calculations in clinical research. CRC press." For a software implementation, dunno what software you use, but if Stata, I suggest ...

Only top voted, non community-wiki answers of a minimum length are eligible