# (When) Does simulating a larger sample from a small sample yield better results?

I am trying to classify a set of $p$ predictors into 5 classes. But my sample size $n$ is rather small, so I fear I won't get a very robust estimate.

Now an idea would be to subset my data for each of the 5 response classes, and simulate more data in each class. I would e.g. assume a multivariate normal distribution for the predictors, and then (I think some people call this a parametric bootstrap) estimate $\mu$ and $\Sigma$ from the multivariate normal, and with those parameters then simulate 10000 new observations in this class. I would do this for every class and then have 50000 additional observations.

Given that I simulate the data from only the knowledge I have from my small sample, is there anything I can gain with this procedure? Robustness yes, but I will certainly not get more information, right?

Does this approach make sense at all? Is there maybe already a better way to solve this problem?

You are falling in the logical mistake of Baron Munchausen:

http://www.lhup.edu/~dsimanek/museum/themes/BaronMunch.jpg

You cannot create more information than that provided by the sample, unless you collect a larger sample. If you simulate more observations from the estimated model $f(\cdot;\hat{\theta})$, then, the more samples you simulate, the more similar to $\hat{\theta}$ the new estimators will be.

Example

set.seed(1234)
samp    <- rnorm(10)
m.samp  <- mean(samp)
s.samp  <- sqrt(var(samp))
b.samp  <- rnorm(1000000,m.samp,s.samp)
mb.samp <- mean(b.samp)
sb.samp <- sqrt(var(b.samp))
# Absolute errors
abs(m.samp-mb.samp)
abs(s.samp-sb.samp)


Also, by adding tons of simulated observations to the real ones, you would induce an artificial precision in the estimation. One of the goals of statistics is precisely related to this: quantifying the uncertainty about a phenomenon. If you require more precision, then you will need more data (which is a common issue in real problems).

• (+1) Welcome to the site, James. If you are the James Joyce I'm thinking of (not the Irish novelist), it's very nice to have you here. I have even seen trained statisticians make this mistake on several occasions, particularly when applying the bootstrap or some variation of crossvalidation. Cheers. – cardinal Mar 13 '13 at 12:05
• Thanks for your response! So to sum up, the answer is "No, it never makes sense"? – Alexander Engelhardt Mar 13 '13 at 13:56