54

I would like to list another intuitive example. Suppose I tell you I can predict the outcome of any coin flip. You do not believe and want to test my ability. You tested 5 times, and I got all of them right. Do you believe I have the special ability? Maybe not. Because I can get all of them right by chance. (Specifically, suppose the coin is a fair coin, ...


51

I agree with many of the other answers here but think the statement is even worse than they make it out to be. The statement is an explicit version of an implicit claim in many shoddy analyses of small datasets. These hint that because they have found a significant result in a small sample, their claimed result must be real and important because it is '...


42

If the population is known to be normal, a 95% confidence interval based on a single observation $x$ is given by $$x \pm 9.68 \left| x \right| $$ This is discussed in the article "An Effective Confidence Interval for the Mean With Samples of Size One and Two," by Wall, Boen, and Tweedie, The American Statistician, May 2001, Vol. 55, No.2. (pdf)


39

[I'll assume from the discussion in your question that you're happy to accept as fact that if $Z_i, i=1,2,\ldots,k$ are independent identically distributed $N(0,1)$ random variables then $\sum_{i=1}^{k}Z_i^2\sim \chi^2_k$.] Formally, the result you need follows from Cochran's theorem. (Though it can be shown in other ways) Less formally, consider that if ...


37

Wayne has addressed the "30" issue well enough (my own rule of thumb: mention of the number 30 in relation to statistics is likely to be wrong). Why numbers in the vicinity of 1000 are often used Numbers of around 1000-2000 are often used in surveys, even in the case of a simple proportion ("Are you in favor of $<$whatever$>$?"). This is done so ...


35

"It's counterintuitive, but the smaller the sample size of a clinical test, the more significant its results are. The differences in a sample of 20 people may be more significant than in a sample of 10,000 people. If we need such a sample, there is a risk of being wrong. With 10,000 people, when the differences are small, sometimes they don't exist." I ...


29

Think about it in terms of proportions. Let's say that preferring an orange is a success, while preferring an apple is a failure. So your mean success rate is $\mu = \frac{\text{# of sucesses}}{n}$ or in this case .6 The standard error of this quantity is estimated to be $\sqrt{\frac{\mu(1-\mu)}{n}}$. For a small sample size (i.e. 10), the standard error is ...


28

Sure there is. Use a Bayesian paradigm. Chances are you have at least some idea of what $\mu$ could possibly be - for instance, that it physically cannot be negative, or that it obviously cannot be larger than 100 (maybe you are measuring the height of your local high school football team members in feet). Put a prior on that, update it with your lone ...


25

The short answer is that your conjecture is true when and only when there is a positive intra-class correlation in the data. Empirically speaking, most clustered datasets most of the time show a positive intra-class correlation, which means that in practice your conjecture is usually true. But if the intra-class correlation is 0, then the two cases you ...


24

(I think the phrasing is deliberately provocative.) If you have 10 observations and want to show that their mean is not zero, it will have to be quite a bit different from 0 if you want to have any kind of chance (power) of detecting the difference. If you have a trillion observations and want to show that their mean is not 0, the mean could be just a tiny ...


22

Statistical tests do not make assumptions about sample size. There are, of course, differing assumptions with various tests (e.g., normality), but the equality of sample sizes is not one of them. Unless the test used is inappropriate in some other way (I can't think of an issue right now), the type I error rate will not be affected by drastically unequal ...


21

Consider tossing a coin which you suspect may come up heads too often. You perform an experiment, followed by a one tailed hypothesis test. In ten tosses you get 7 heads. Something at least as far from 50% could easily happen with a fair coin. Nothing unusual there. If instead, you got 700 heads in 1000 tosses, a result at least as far from fair as that ...


21

You should rarely trust any single significant result. You didn't say why you were using a one-tailed instead of a two-tailed test, so hopefully you have a good reason for doing so other than struggling to be able to claim a statistically significant outcome! Setting that aside, consider the following from p. 261 of Sauro, J., & Lewis, J. R. (2016). ...


21

TL;DR: The minimum sample size per cluster in a mixed-effecs model is 1, provided that the number of clusters is adequate, and the proportion of singleton cluster is not "too high" Longer version: In general, the number of clusters is more important than the number of observations per cluster. With 700, clearly you have no problem there. Small cluster ...


19

I typically use a 15:1 rule (ratio of min(events, non-events) to number of candidate parameters in the model). More recent work found that for a more rigorous validation 20:1 is needed. More information may be found in my course handouts linked from http://hbiostat.org/rms, in particular an argument for a minimum sample size of 96 just to estimate the ...


19

I don't prefer to think of this as a power issue, but rather ask the question "how large should $n$ be so that the apparent $R^2$ can be trusted"? One way to approach that is to consider the ratio or difference between $R^2$ and $R_{adj}^{2}$, the latter being the adjusted $R^2$ given by $1 - (1 - R^{2})\frac{n-1}{n-p-1}$ and forming a more unbiased ...


19

The question you are asking is different from "convergence diagnostics". Lets say you have run all convergence diagnostics(choose your favorite(s)), and now are ready to start sampling from the posterior. There are two options in terms of effective sample size(ESS), you can choose a univariate ESS or a multivariate ESS. A univariate ESS will provide an ...


19

Some books state a sample size of size 30 or higher is necessary for the central limit theorem to give a good approximation for $\bar{X}$. This common rule of thumb is pretty much completely useless. There are non-normal distributions for which n=2 will do okay and non-normal distributions for which much larger $n$ is insufficient - so without an explicit ...


19

The standard deviation is as applicable here as anywhere else: it gives useful information about the dispersion of the data. In particular, the sd divided by the square root of the sample size is one standard error: it estimates the dispersion of the sampling distribution of the mean. Let's calculate: $$3.2\% / \sqrt{10000} = 0.032\% = 0.00032.$$ That's ...


17

In an attempt to figure out the minimum number of groups for a multilevel model I looked at the book Data Analysis Using Regression and Mulitilevel/Hierarchical models by Gelman and Hill (2007). They appear to address this topic in Chapter 11, Section 5 (page 247) where they write that when there are < 5 groups then multilevel models typically add little ...


17

You surely found the very similar question: Choice of K in K-fold cross-validation ? (Including the link to Ron Kohavi's work) If your sample size is already small I recommend avoiding any data driven optimization. Instead, restrict yourself to models where you can fix hyperparameters by your knowledge about model and application/data. This makes one of the ...


17

Note that at each observation position ($i=1, 2, ..., n$) we can choose any of the $n$ observations, so there are $n^n$ possible resamples (keeping the order in which they are drawn) of which $n!$ are the "same sample" (i.e. contain all $n$ original observations with no repeats; this accounts for all the ways of ordering the sample we started with). For ...


17

This concept is a consequence of the law of large numbers. From Wikipedia, According to the law, the average of the results obtained from a large number of trials should be close to the expected value, and will tend to become closer as more trials are performed. Results from a small sample may be farther from the expected value than that from a larger ...


16

This is a question that is not asked often enough. In frequentist statistics we tend to fix $\alpha$ by convention. Then as $n\rightarrow\infty$ the type II error $\rightarrow 0$ (i.e., power $\rightarrow 1$) even though we also have the luxury for large $n$ of not allowing so many false positives had we chosen differently. The result of this convention ...


16

In theory if all the assumptions of the t-test are true then there's no problem with a small sample size. In practice there are some not-quite-true assumptions which we can get away with for large sample sizes but they can cause problems for small sample sizes. Do you know if the underlying distribution is normally distributed? Are all the samples ...


16

Imagine yourself to be in a situation where you're doing many similar tests, in a set of circumstances where some fraction of the nulls are true. Indeed, let's model it using a super-simple urn-type model; in the urn, there are numbered balls each corresponding to an experiment you might choose to do, some of which have the null true and some which have the ...


15

The question of "significantly" different always, always presupposes a statistical model for the data. This answer proposes one of the most general models that is consistent with the minimal information provided in the question. In short, it will work in a wide array of cases, but it might not always be the most powerful way to detect a difference. Three ...


14

For what it's worth, I did a bit of a simulation study to look at the stability of the variance estimate for a relatively simple LMM (using the sleepstudy dataset available through lme4). The first way generates all possible subject combinations for ngroups number of subjects, and refits the model for each possible combination. The second takes several ...


14

A small simulation exercise to illustrate whether the answer by @soakley works: # Set the number of trials, M M=10^6 # Set the true mean for each trial mu=rep(0,M) # Set the true standard deviation for each trial sd=rep(1,M) # Set counter to zero count=0 for(i in 1:M){ # Control the random number generation so that the experiment is replicable set.seed(i)...


14

No. Many data sets can yield the same mean, SD and n. In the graph above, the three data sets on the left of each graph all share the same mean, SD and n. So do the three data sets on the right of each graph. This is Figure 1 from: Weissgerber, T.L., Milic, N.M., Winham, S.J., and Garovic, V.D. (2015). Beyond bar and line graphs: time for a new data ...


Only top voted, non community-wiki answers of a minimum length are eligible