50

I agree with many of the other answers here but think the statement is even worse than they make it out to be. The statement is an explicit version of an implicit claim in many shoddy analyses of small datasets. These hint that because they have found a significant result in a small sample, their claimed result must be real and important because it is '...


35

"It's counterintuitive, but the smaller the sample size of a clinical test, the more significant its results are. The differences in a sample of 20 people may be more significant than in a sample of 10,000 people. If we need such a sample, there is a risk of being wrong. With 10,000 people, when the differences are small, sometimes they don't exist." I ...


22

(I think the phrasing is deliberately provocative.) If you have 10 observations and want to show that their mean is not zero, it will have to be quite a bit different from 0 if you want to have any kind of chance (power) of detecting the difference. If you have a trillion observations and want to show that their mean is not 0, the mean could be just a tiny ...


11

Can you confirm that it is a FALSE statement in statistics I think the statement is phrased poorly. In this context, the word "significant" seems to have the flavor of "importance". Difference detected in smaller datasets are not somehow more important or meaningful by virtue of being detected in small datasets. Rather, differences detected in small ...


9

Coronaviruses are a group of viruses. According to the Wiki, coronaviruses were discovered in the 1960s. https://en.wikipedia.org/wiki/Coronavirus So people could have searched for "coronavirus" long before the current outbreak. Case in point, Google trends for year 2018: And here for reference Google Ngram, indeed showing the term first appeared in the ...


8

The quote in question seems to originate from marianne.net (in French) and, as it stands, is definitely wrong. But, as Demetri and Dave pointed out, with some language bending there might be some truth to it. In my understanding, Prof. Raoult confuses significance and effect size. In a small sample, the effect size has to be large (i.e. of practical ...


7

I have concerns about involving 250 predictors when you have 16 samples. However, let's set that aside for now and focus on cross-validation. You don't have much data, so any split from the full set to the training and validation set is going to result in really very few observations on which you can train. However, there is something called leave-on-out ...


7

For the purposes of a hypothesis test, there are two related approaches to finding an optimal sample size that are viable if you're willing to assume bivariate normality. Power To estimate minimal sample size at a given confidence level ($1-\alpha$) and power ($1-\beta$), we can use a modification of the equation for calculating the power of a Pearson ...


6

I think it's been asked before. It's useful to realize that, without a prespecified sample size and alpha level, the $p$-value is just a measure of the sample size you ultimately wind up with. Not appealing. An approach I use is this: at what sample size would a 0.05 level be appropriate? Scale accordingly. For instance, I feel the 0.05 level is often suited ...


5

A smaller sample size is not better. A small sample size needs a more significant* result if you want to draw a conclusion from it. Let's consider some results and their interpretation: If your drug cures 30% of 10 people, the percentage of the general population cured could be anywhere between around 0% and 65% of people. If your drug cures 30% of 10000 ...


5

All sample size calculations are built on top of a proposed inference that will be made from the data. This might be a confidence interval for an unknown parameter, or a hypothesis test for a set of hypotheses, or a Bayesian posterior inference, etc. Whatever the inference being made, there will be some appropriate measure of how "accurate" the inference ...


5

A typical way to use the sample size from the normal assumption would be to approximate the number of degrees of freedom in the t-distribution. Say your sample size calculation, using the normal distribution, gives a required sample size of 60. You would then use $n=60$ and $df=59$ in the sample size calculation with the $t_{df}$ distribution. Yes, you ...


4

A classic way to proceed is to determine the difference that you wish to be able to detect at a given combination of Type I error and Type II error, and then design a study with a large enough sample size to meet your requirements, given the variability you expect in your measurements. All of the "playing" you are doing is legitimate in some sense. In ...


4

If you find a single mismatched pair, you know the data sets are not matched. It's impossible both to find a mismatched pair and for the datasets to be perfectly matched. So any test that relies on finding any mismatched pairs will have a false alarm rate (i.e., alpha) of 0. The power of any test depends on the true "effect". The datasets would be ...


4

This question is sufficiently broad that a comprehensive answer would require a simulation study (some key parts of which have undoubtedly been done), going far beyond our usual style of answers. A Welch 2-sample t test can't be exactly as good as a pooled t test if we know the two populations have the same variance. (1) A Welch test at "level 5%" with $...


4

There are some problems with your approach: Mean of ranks (ordinal data), may or may not make sense, depending on your application. Is it sensible to treat a bank of rank 15 as 15 times a bank of rank 1? I.e. is the mean between a bank of rank 1 and one of rank 15 (given equal share) a bank of rank 8? You should think about if your estimate makes sense at ...


4

There are a few things that are true, and worth understanding for where the confusion might slip in. First, it is possible to get high levels of confidence from small samples, so long as the effect is sufficiently pronounced. For example, a treatment that goes from 10% control recovery to 90% experimental recovery will show up with a very good score even if ...


4

It's your power. Note in that first calculator link you can set the power downward to 60%, and that lowers the sample needed to 3310. If you could lower the power there to 50%, you would likely get 2725. When you make that significance test using the second calculator, you are at a critical point. If you had one less positive result, you wouldn't be ...


4

The way you describe it, i.e. estimating the effect size based on the data and then checking post-hoc the power based on the estimated effect size is how post-hoc tests are often done. Unfortunately, it is NOT how post-hoc power estimates should be done. See for example O'Keefe (2010). The reason for this is, that a post-hoc power analysis of this type ...


3

I would say that the sample sizes are too small. In each group (Black and White), there will be considerable diversity. There are subpopulations. I think that having fewer than 80 subjects in each group means that the samples will miss some of those subpopulations. Even if they get statistically significant results, I am skeptical that enough subpopulations ...


3

"What I find truly interesting is that the total theoretical density doesn't seem to depend on the underlying density function!" For continuous distributions this is not at all surprising. Consider transforming any continuous variate by its cdf $U=F_X(X)$ (i.e. the probability integral transform), which yields a (standard) uniform. When you do this, you ...


3

Effect size has many possible meanings, as the Wikipedia page shows. For G*Power, it means the following unitless ratio: The difference of the means between the lowest group and the highest group over the common standard deviation is a measure of effect size. So your value of 0.5 for effect size is not correct; that's the difference of A1C units between ...


3

If you want the width of your confidence interval to have length no larger than 1 (0.5 on each side) then let's do some simple math. The radius of a confidence interval, as a function of the sample size, is $$ \dfrac{1.96 \cdot \sigma}{\sqrt{n}}$$ I'm using the z-statistic here instead of a t-statistic because in the limit, the two are equivalent (I'm ...


3

The classical way of testing whether two (R)MSEs would be the Diebold-Mariano test. However, this is an asymptotic test, and you have a rather small sample size, so something else would be in order. Let's simulate some data. I'll use R. set.seed(1) group_1 <- rnorm(40,mean=5.5,sd=2.8) group_2 <- rnorm(5,mean=12.4,sd=4.4) Now, the first test to run ...


3

I am not sure if the title and the body of your question are asking about the same thing, but I will try to answer in a general way to cover both. \begin{aligned} \text{AIC}&=-2(\text{loglik}-p) \\ &=-2(n\times \text{avg(loglik)}-p) \end{aligned} where $n$ is the sample size used for estimation, $p$ is the number of the model's degrees of freedom ...


3

I think the best walkthrough of how to sample size plan using lavaan and simsem I have ever read was in a very short and very accessible text called Latent Variable Modeling Using R: A Step-by-Step Guide by A. Alexander Beaujean. His examples are admittedly relatively simple, but they should provide an excellent basis to start from when thinking about a ...


3

The issue here does not appear to be due to imbalanced sample sizes per se, but rather, non-uniformity of the p-value seems to be occurring due to using very small sample sizes for one of the samples. Below I show high-resolution histograms showing simulated p-values for a comparison of one sample with $n_X = 100$ standard normal values and another sample ...


3

The problem asks for the sample size required so that the confindence interval has width 4. The interval is symmetric around $\mu$, and the distance from $\mu$ to each edge of the interval is $1.96\left(\frac{4}{\sqrt{n}}\right).$ The total width is therefore two times this. So, to make the confidence interval have width 4, $n$ must satisfy the equation $1....


3

Consistency does not depend on sample size; it judges if an estimator formula converges its target value in probability. This means we take the limit as $n$ goes to $\infty$ and get rid of the sample size. If you want to somehow quantify the risk you take at your specific sample size, you can look at the variance of your estimator.


3

Nick is technically right. But note that if you assume that you have a series of random samples from the underlying population, larger sample sizes (larger $n$) cause $Var(\hat\beta)$ to decrease asymptotically. To see this, note that $\frac{1}{n} \sum (x_i - \bar{x})^2 \overset{p}{\to} Var(x)$ so $\sum (x_i - \bar{x})^2 \overset{p}{\to} n \,Var(x) = \infty$...


Only top voted, non community-wiki answers of a minimum length are eligible