12
$\begingroup$

I have references that advised to consider a sample size of minimum 20 for data fitting distribution.

Is there any sense in this?

thanks

$\endgroup$
  • 3
    $\begingroup$ In general no, in specific situations maybe. Do you have the references and what are your goals ? $\endgroup$ – image_doctor Apr 18 '13 at 13:44
  • 1
    $\begingroup$ I'd agree with @image_doctor - there's no general reason why 20 observations should be regarded as a minimum; it might be the case in very particular circumstances. $\endgroup$ – Glen_b -Reinstate Monica Apr 18 '13 at 14:28
  • 3
    $\begingroup$ A single value will work in some practical cases, such as when the distribution is known (with certainty) to be Poisson and the observation is a large count. Not only does that allow the distribution to be fit, it also permits the likely error in the estimate of its parameter to be assessed. $\endgroup$ – whuber Apr 18 '13 at 14:31
  • 1
    $\begingroup$ I believe I've also read that for normal distributions, 30 is the rule-of-thumb. Seems to me it was something related to how close a Student's t with 30 degrees of freedom is to the normal. But it's just a rule of thumb. It's not magic in the same sense that the value of $e$ is. $\endgroup$ – Wayne Apr 18 '13 at 15:17
  • 1
    $\begingroup$ Yes, 20 is a magic number: en.wikipedia.org/wiki/Magic_number_%28physics%29 $\endgroup$ – Bitwise Apr 18 '13 at 17:20
13
$\begingroup$

So much of this depends on the expected distribution and what your research question is. As a rule of thumb, you should be wary of rules of thumb. If you know the expected distribution, run some simulations of different sizes and determine how often the sample simulations reflect the actual distribution. This should give you some guidance as the final required sample size.

$\endgroup$
  • $\begingroup$ +1 for steering clear of extreme and dogmatic statements. $\endgroup$ – whuber Apr 18 '13 at 16:16
  • 1
    $\begingroup$ +1 in part due to "As a rule of thumb, you should be wary of rules of thumb." $\endgroup$ – Wolfgang Apr 19 '13 at 7:13
7
$\begingroup$

I thought the sample size magic number is 1,000. That's what most US national polls have, to produce the margin of error of about 3%: $$ z_{0.975}\sqrt{0.5\cdot0.5/1000} = 1.96 \cdot 0.158 = 0.031 $$ In reality, the effective sample sizes are lower than 1,000, more like 700 or so, due to unequal probability of selection and non-response adjustments, leading to the margin of error of 3.7%.

With just 20 observations, you technically cannot get very high values of skewness and kurtosis (normalized by the sample standard deviations, of course): $$ |\mbox{skewness}| \le \frac{n-2}{\sqrt{n-1}} = 4.58, |\mbox{kurtosis}| \le \frac{n^2 - 3n + 3}{n-1} = 18.05. $$ If you are fitting a distribution by the method of moments, you obviously cannot fit say a lognormal distribution with a reasonably typical variance of logs equal to 1 (income distributions in moderate-to-high income inequality countries; US, Brazil, South Africa, Russia all have higher variance of log-incomes), as it has a surprisingly large kurtosis of 111. Of course it would be silly to fit a lognormal distribution by the method of moments, but I just wanted to show that some real world distributions will likely be more complicated than what can be described with 20 observations.

Another view on distribution fitting can be taken through kernel density estimation: for the sample of size $n=20$, the most popular rule gives the bandwidth of $$ h=1.06 \hat\sigma n^{-1/5}=0.58\hat\sigma $$ which effectively spans the whole distribution using the Gaussian kernel. In other words, most samples of size 20 will look normal if you run kernel density estimation across them, unless they clearly have a notable kurtosis (which would then mean that there are some outlying observations that will show up as separate bumps on the kernel density plot).

$\endgroup$
  • $\begingroup$ I don't follow the relevance of limits on the moments of samples. Of course you can get extremely high estimates of skewness and kurtosis in a sample. Try it: when $m$ is the sample mean and $s$ the sample SD, then the estimated skewness when matching moments in a lognormal distribution is $(s/m)(3 + (s/m)^2)$. Generate a sample of $20$ from a distribution with a large geometric SD ($2$ will work) and you'll get huge estimates of skewness. So what if the raw sample skewness is small? $\endgroup$ – whuber Apr 18 '13 at 16:12
  • 1
    $\begingroup$ Obligatory link to "Power-of-Ten Syndrome" in the context of using 1000 as a sample size (in the context of stats related to programming, but it applies elsewhere): zedshaw.com/essays/programmer_stats.html $\endgroup$ – Gary S. Weaver Apr 18 '13 at 16:26
  • 1
    $\begingroup$ @whuber, you are way too smart, no offense. Most people would compute skewness as the moment of data, not through a parametric assumption as you just did. Now, if I said "Suppose you are fitting a Pearson distribution by method of moments" -- would this argument be relevant, then? $\endgroup$ – StasK Apr 18 '13 at 19:16
  • $\begingroup$ Yes, it would be relevant in many cases. I followed your suggestion to fit a lognormal distribution using the method of moments and obtained a large skewness--no surprise there. This is because I matched only the first two moments, letting the third be what it may. If I were to follow the same procedure with any distribution family of two or fewer parameters that allows arbitrarily large third moments, I think I would see the same phenomenon. With the Pearson family, which has more than two parameters, we would likely try to match the empirical skewness, thereby limiting its value. $\endgroup$ – whuber Apr 18 '13 at 19:48
2
$\begingroup$

Nope. Not remotely.

Think about it like this: if you had a billion-dimensional space (humanity) and you pulled 20 samples out using whatever method (20 people) could you use information gained them to understand reasonably well every person on the planet? Not remotely. There are 100 billion stars in the Milky Way galaxy. By picking (randomly) 20 of them can you understand all of galactic astronomy? No way.

In a 1-d space there are some heuristics, mostly valid rules of thumb that can help, that describe how many measurements you want to take. They include varying degrees of utility and justification but are in some sense more well defended than "20". They include "5 measurements per variable in your fit equation", "at least 35 samples of a Gaussian density function", and "at least 300 samples of a binomial function". Real statisticians and not a nerd-bomber like me will be able to associate particular confidence intervals and uncertainties from first principles and without a calculator.

If you use the rule of "5 measurements per parameter in your fit equation" and you want to fit the cumulative density of a 2 dimensionally curved bi-cubic surface in terms of distribution of heights you are going to have an underlying system that is $ \int {\int {\frac {a_3{r^3}+a_2{r^2}+a_1r+a_0}{a_1r+a_0}}} dr $, a ratio of a 5th order polynomial to a cubic. It will have 6+4=10 coefficients. If you are trying to fit your 10 parameter values using 2 measurements per parameter, or by using 20 measurements, then you will be violating this heuristic. This heuristic recommends a minimum of 10*5=50 measurements.

Please remember "best" is a meaningless idea without having a "measure of goodness". What is the best path? If you are going to your doom, maybe an extremely long and pleasant one. If you are going to your own coronation, maybe a short and magnificent one. If you are walking through the desert, a cool shady one. What is the "best" number of samples? It is so stunningly dependent on your problem that it cannot begin to be answered with authority before that. All of them? As many as you can? Those only make a little sense. Yes it is like being partially dead or pregnant. Being partially nonsensical is a consequence of a very under-defined problem.

If you are trying to accurately forecast airflow over a plane? You might need several million measurements to get into the ball-park. If you want to know how tall you are, one or two might do the job.

This does not bring up the important points of "spanning the space" and "sampling at locations that minimize variance in parameter estimates" but the the question suggested a more freshman-level answer would be relevant. These things require knowing more about the nature of the problem before they can be implemented.

Note: edited to improve per suggestions.

$\endgroup$
  • 1
    $\begingroup$ You seem to have read "minimum" in the question as "maximum" or "sufficient." Nothing you have written appears to contradict a minimum-of-20 rule of thumb. $\endgroup$ – whuber Apr 18 '13 at 16:16
  • 2
    $\begingroup$ @whuber, I work among folks who think extra measurements are expensive, and if I supply them a "minimum number of samples" they do not think of it as an inequality where the potential number of samples is greater than that. They think of it as the boundary for an optimization problem of minimizing cost, and try to operate at only that value. It is a product of my environment. $\endgroup$ – EngrStudent Apr 19 '13 at 0:53
1
$\begingroup$

Perhaps for the context where you're carrying out t-tests or ANOVAR - a pretty common context in basic statistical applications - it's around the sample size you need for each group in order to be able to have much confidence in each group's mean's being approximately normally distributed (according to the central limit theorem) when the distribution can be assumed to be more or less unimodal & not extremely peaky. Twenty & not nineteen or twenty-one because it's a round number.

$\endgroup$
0
$\begingroup$

Check Russ Lenth's Power and Sample Size Page for some articles on the subject (in the Advice section in the middle of the page).

The minimum number of individuals in your sample varies wildly according to population size, the number of dimensions (if you are dividing the data in categories) and measures (if you are taking continuous measures about the sample indivifuals) you are taking, the size of your universe, the technique of analysis you intend to use (this is a very important point - technique is defined during the planning of the study or during experimental design, never after), and complexity shown by previous studies.

And 20 is not enough for any serious research outside the subjects of "rare diseases" and "experimental psychology" (psych as Popper defined in his work).

Refining the answer based on comments below:

And 20 is not enough for any serious research outside the subjects of "rare diseases" and "experimental psychology" (psych as Popper defined in his work) that involves fitting a probability distribution.

And no, you should not keep poisoning people to get to a large sample size. Common Sense and Sequential Tests order you to stop.

$\endgroup$
  • 3
    $\begingroup$ I think it's too extreme to give a blanket statement that a sample of 20 is "not enough for any serious research." This contradicts your earlier statements that the appropriate sample size varies with purpose, population, and so on. In some cases one falsifying result is enough to kill an entire theory. $\endgroup$ – whuber Apr 18 '13 at 16:14
  • 1
    $\begingroup$ Case studies and qualitative research can do fine with 1-5 participants. $\endgroup$ – Behacad Apr 18 '13 at 16:24
  • $\begingroup$ Ok, add "case studies" and "focus groups" to the list :) These are included in what I said Popper called "experimental psychology" tho. $\endgroup$ – Lucas Gallindo Apr 18 '13 at 16:34
  • 4
    $\begingroup$ You had then better add astronomy, medicine, biology, chemistry, ... In other words, it's just as bad to assert that 20 is "not enough" as it is to assert that it's good enough. Actually, it's probably worse. Imagine a food safety trial in which the first eight subjects who were administered a nutritional supplement died of unanticipated side effects. Would you advocate continued testing on the basis of your "20 is not enough" statement? $\endgroup$ – whuber Apr 18 '13 at 16:42

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.