(When) Does simulating a larger sample from a small sample yield better results?

Question

I am trying to classify a set of $p$ predictors into 5 classes. But my sample size $n$ is rather small, so I fear I won't get a very robust estimate.

Now an idea would be to subset my data for each of the 5 response classes, and simulate more data in each class. I would e.g. assume a multivariate normal distribution for the predictors, and then (I think some people call this a parametric bootstrap) estimate $\mu$ and $\Sigma$ from the multivariate normal, and with those parameters then simulate 10000 new observations in this class. I would do this for every class and then have 50000 additional observations.

Given that I simulate the data from only the knowledge I have from my small sample, is there anything I can gain with this procedure? Robustness yes, but I will certainly not get more information, right?

Does this approach make sense at all? Is there maybe already a better way to solve this problem?

Answer 1

You are falling in the logical mistake of Baron Munchausen:

http://www.lhup.edu/~dsimanek/museum/themes/BaronMunch.jpg

You cannot create more information than that provided by the sample, unless you collect a larger sample. If you simulate more observations from the estimated model $f(\cdot;\hat{\theta})$, then, the more samples you simulate, the more similar to $\hat{\theta}$ the new estimators will be.

Example

set.seed(1234)
samp    <- rnorm(10)
m.samp  <- mean(samp)
s.samp  <- sqrt(var(samp))
b.samp  <- rnorm(1000000,m.samp,s.samp)
mb.samp <- mean(b.samp)
sb.samp <- sqrt(var(b.samp))
# Absolute errors
abs(m.samp-mb.samp)
abs(s.samp-sb.samp)

Also, by adding tons of simulated observations to the real ones, you would induce an artificial precision in the estimation. One of the goals of statistics is precisely related to this: quantifying the uncertainty about a phenomenon. If you require more precision, then you will need more data (which is a common issue in real problems).

Answer 2

This sounds like "data augmentation", which is often applied when training neural networks (perhaps because these classifiers have many parameters, and you want to avoid over-fitting to irrelevant properties of the limited training set).

For example, if you are classifying images by subject, then to enlarge your training set you might try slight rotations, translations, rescaling or adding noise. (It probably involves a level of prior knowledge about the problem domain, to tell which transformations will simulate an equally-plausible sample of the same class.)