I am trying to classify a set of $p$ predictors into 5 classes. But my sample size $n$ is rather small, so I fear I won't get a very robust estimate.

Now an idea would be to subset my data for each of the 5 response classes, and simulate more data in each class. I would e.g. assume a multivariate normal distribution for the predictors, and then (I think some people call this a parametric bootstrap) estimate $\mu$ and $\Sigma$ from the multivariate normal, and with those parameters then simulate 10000 new observations in this class. I would do this for every class and then have 50000 additional observations.

Given that I simulate the data from only the knowledge I have from my small sample, is there anything I can gain with this procedure? Robustness yes, but I will certainly not get more information, right?

Does this approach make sense at all? Is there maybe already a better way to solve this problem?


2 Answers 2


You are falling in the logical mistake of Baron Munchausen:


You cannot create more information than that provided by the sample, unless you collect a larger sample. If you simulate more observations from the estimated model $f(\cdot;\hat{\theta})$, then, the more samples you simulate, the more similar to $\hat{\theta}$ the new estimators will be.


samp    <- rnorm(10)
m.samp  <- mean(samp)
s.samp  <- sqrt(var(samp))
b.samp  <- rnorm(1000000,m.samp,s.samp)
mb.samp <- mean(b.samp)
sb.samp <- sqrt(var(b.samp))
# Absolute errors

Also, by adding tons of simulated observations to the real ones, you would induce an artificial precision in the estimation. One of the goals of statistics is precisely related to this: quantifying the uncertainty about a phenomenon. If you require more precision, then you will need more data (which is a common issue in real problems).

  • $\begingroup$ (+1) Welcome to the site, James. If you are the James Joyce I'm thinking of (not the Irish novelist), it's very nice to have you here. I have even seen trained statisticians make this mistake on several occasions, particularly when applying the bootstrap or some variation of crossvalidation. Cheers. $\endgroup$
    – cardinal
    Commented Mar 13, 2013 at 12:05
  • 1
    $\begingroup$ Thanks for your response! So to sum up, the answer is "No, it never makes sense"? $\endgroup$ Commented Mar 13, 2013 at 13:56

This sounds like "data augmentation", which is often applied when training neural networks (perhaps because these classifiers have many parameters, and you want to avoid over-fitting to irrelevant properties of the limited training set).

For example, if you are classifying images by subject, then to enlarge your training set you might try slight rotations, translations, rescaling or adding noise. (It probably involves a level of prior knowledge about the problem domain, to tell which transformations will simulate an equally-plausible sample of the same class.)


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.