5
$\begingroup$

I have a list of 6,500 or so medical treatments, of which I have classified 700 or so as involving a physician or not. I am interested in both the specific question of how to calculate whether 700 observations is sufficient, and the general question about how one does power calculations in text mining.

A typical treatment might have 3-10 words describing the intervention (e.g. "Coronary artery bypass graft surgery for left main artery disease"). My training dataset adds to that a binary variable indicating whether a physician is involved in the process or not. I want to predict this binary variable on all the remaining observations.

I can classify more if necessary, but obviously the time and money cost increases to do so. I am curious how I might know when to stop creating a training dataset.

I assume that a "power calculation" would involve:

  • Characteristics of the text (number of words per observation, prevalence of sparse words, overlap of the words between each of the binary categories, etc.)
  • Desired prediction accuracy
  • Algorithm used

Is there a formal method for this? I'm using RTextTools if that helps.

Improve this question
$\endgroup$
2
  • $\begingroup$ I'm not sure whether you're using the term "power" in the standard statistical sense--namely, the probability of finding an effect statistically significant based on a sample, assuming that effect exists in the larger population. Unless I'm way off, power is always calculated with respect to a particular procedure--T-test, logistic regression, etc. Maybe that's what you meant by "algorithm." What procedure are you planning to use? $\endgroup$
    – rolando2
    Feb 12, 2012 at 20:16
  • $\begingroup$ I am using power in that sense. I realized after I wrote this that one generally thinks of power calculations as frequentist, and many ML techniques have Bayesian analogues. So maybe power calculations don't make any sense in this world and I should just classify a few times and see whether the results are good enough. $\endgroup$
    – Ari B. Friedman
    Feb 12, 2012 at 20:35

1 Answer 1

Reset to default
2
$\begingroup$

To create a reliable trained classifier in RTextTools, you first compare its accuracy to data that has been manually classified. The create_corpus() function lets you do this by specifying the trainSize, testSize, and then setting the "virgin" flag to FALSE. This indicates that the test data has already been manually classified and is not "virgin" unclassified data, and RTextTools should compare its results against yours.

There are a variety of analytics to see how good your classifier is-- for example, the @ensemble_summary. The n-ensemble recall accuracy score shows you how accurately RTextTools classified the data in the test set. Another analytic is the @algorithm_summary which shows the precision, F-score, recall, and overall accuracy broken down by category.

Randomly sample your training and testing data over several iterations, and if these scores are consistently high you should have a reliable classifier.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.