I have a list of 6,500 or so medical treatments, of which I have classified 700 or so as involving a physician or not. I am interested in both the specific question of how to calculate whether 700 observations is sufficient, and the general question about how one does power calculations in text mining.
A typical treatment might have 3-10 words describing the intervention (e.g. "Coronary artery bypass graft surgery for left main artery disease"). My training dataset adds to that a binary variable indicating whether a physician is involved in the process or not. I want to predict this binary variable on all the remaining observations.
I can classify more if necessary, but obviously the time and money cost increases to do so. I am curious how I might know when to stop creating a training dataset.
I assume that a "power calculation" would involve:
- Characteristics of the text (number of words per observation, prevalence of sparse words, overlap of the words between each of the binary categories, etc.)
- Desired prediction accuracy
- Algorithm used
Is there a formal method for this? I'm using RTextTools if that helps.