I need to classify large numbers of short answer, free response data from a study with a between-group design.

In order to reduce the amount of manual labor costs, I was thinking of manually coding a small sample set, running the rest of the responses through an SVM classifier, and then coding a random sample of the SVM classifier to obtain a classical statistical measures of the automatically coded data-set.

The original, and overly verbose, title to this question was, "Is applying random sampling to output from a machine learning classifier a statistically valid way to calculate confidence intervals?"

I have already done a conceptual sanity check with a friend of mine who worked with machine-learning algorithms and atmospheric modeling, but I wanted to run it past some real statisticians before I start basing my workflow around this.



In my opinion, you are kind of hinting towards the so-called bootstrap method of deriving confidence intervals, which is sound enough. The wiki says-

Bootstrapping is the practice of estimating properties of an estimator (such as its variance) by measuring those properties when sampling from an approximating distribution. One standard choice for an approximating distribution is the empirical distribution of the observed data.

Here, the output is the approximate distribution of data and randomly sampling from it is conceptually sound enough. See this link


| cite | improve this answer | |
  • $\begingroup$ This is what my atmospheric scientist friend and I had basically come to agreement about, but we didn't have a name for it. I would still like to leave the question up in case some academic with experience doing this wants to chime in. Still, 100% helpful, thanks! $\endgroup$ – Indolering Dec 27 '11 at 21:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.