# Text mining “power calculations”?

I have a list of 6,500 or so medical treatments, of which I have classified 700 or so as involving a physician or not. I am interested in both the specific question of how to calculate whether 700 observations is sufficient, and the general question about how one does power calculations in text mining.

A typical treatment might have 3-10 words describing the intervention (e.g. "Coronary artery bypass graft surgery for left main artery disease"). My training dataset adds to that a binary variable indicating whether a physician is involved in the process or not. I want to predict this binary variable on all the remaining observations.

I can classify more if necessary, but obviously the time and money cost increases to do so. I am curious how I might know when to stop creating a training dataset.

I assume that a "power calculation" would involve:

• Characteristics of the text (number of words per observation, prevalence of sparse words, overlap of the words between each of the binary categories, etc.)
• Desired prediction accuracy
• Algorithm used

Is there a formal method for this? I'm using RTextTools if that helps.

• I'm not sure whether you're using the term "power" in the standard statistical sense--namely, the probability of finding an effect statistically significant based on a sample, assuming that effect exists in the larger population. Unless I'm way off, power is always calculated with respect to a particular procedure--T-test, logistic regression, etc. Maybe that's what you meant by "algorithm." What procedure are you planning to use? – rolando2 Feb 12 '12 at 20:16
• I am using power in that sense. I realized after I wrote this that one generally thinks of power calculations as frequentist, and many ML techniques have Bayesian analogues. So maybe power calculations don't make any sense in this world and I should just classify a few times and see whether the results are good enough. – Ari B. Friedman Feb 12 '12 at 20:35

There are a variety of analytics to see how good your classifier is-- for example, the @ensemble_summary. The n-ensemble recall accuracy score shows you how accurately RTextTools classified the data in the test set. Another analytic is the @algorithm_summary which shows the precision, F-score, recall, and overall accuracy broken down by category.