Is there a rule-of thumb or even any way at all to tell how large a sample should be in order to estimate a model with a given number of parameters?
So, for example, if I want to estimate a least-squares regression with 5 parameters, how large should the sample be?
Does it matter what estimation technique you are using (e.g. maximum likelihood, least squares, GMM), or how many or what tests you are going to perform? Should the sample variability be taken into account when making the decision?
The trivial answer is that more data are always preferred to less data.
The problem of small sample size is clear. In linear regression (OLS) technically you can fit a model such as OLS where n = k+1 but you will get rubbish out of it i.e. very large standard errors. There is a great paper by Arthur Goldberger called Micronumerocity on this topic which is summarized in chapter 23 of his book A Course in Econometrics.
A common heuristic is that you should have 20 observations for every parameter you want to estimate. It is always a trade off between the size of your standard errors (and therefore significance testing) and the size of your sample. This is one reason some of us hate significance testing as you can get an incredibly small (relative) standard error with an enormous sample and therefore find pointless statistical significance on naive tests such as whether a regression coefficient is zero.
While sample size is important the quality of your sample is more important e.g. whether the sample is generalisable to the population, is it a Simple Random Sample or some other appropriate sampling methodology (and have this been accounted for during analysis), is there measurement error, response bias, selection bias, etc.
I've heard two rules of thumb in this regard. One holds that so long as there are enough observations in the error term to evoke the central limit theorem, e.g. 20 or 30, you are fine. The other holds that for each estimated slope one should have at least 20 or 30 observations. The difference between using 20 or 30 as the target number is based on different thoughts regarding when there are enough observations to reasonably evoke the Central Limit Theorem.
I like to use resampling: I repeat whatever method I used with a subsample of the data (say 80% or even 50% of the total). By doing this with many different subsamples, I get a feel for how robust the estimates are. For many estimation procedures this can be made into a real (meaning publishable) estimate of your errors.
It should always be large enough! ;)
All parameter estimates come with an estimate uncertainty, which is determined by the sample size. If you carry out a regression analysis, it helps to remind yourself that the Χ2 distribution is constructed from the input data set. If your model had 5 parameters and you had 5 data points, you would only be able to calculate a single point of the Χ2 distribution. Since you will need to minimize it, you could only pick that one point as a guess for the minimum, but would have to assign infinite errors to your estimated parameters. Having more data points would allow you to map the parameter space better leading to a better estimate of the minimum of the Χ2 distribution and thus smaller estimator errors.
Would you be using a Maximum Likelihood estimator instead the situation would be similar: More data points leads to better estimate of the minimum.
As for point variance, you would need to model this as well. Having more data points would make clustering of points around the "true" value more obvious (due to the Central Limit Theorem) and the danger of interpreting a large, chance flucuation as the true value for that point would go down. And as for any other parameter your estimate for the point variance would become more stable the more data points you have.
