Take the 2-minute tour ×
Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It's 100% free, no registration required.

I am trying to assess the performance of my Bayesian classifier. One measure that I calculate is the false discovery rate (FDR):

FP / (FP + TP), where FP = False Positive and TP = True Positive.

The ratio (and sizes) of my positive and negative training sets do not accurately reflect the estimated 'real' ratios (and sizes). To clarify, I am working with sets of genes that have a certain property. It is estimated that there are approximately 18000 negative (not having that property) and 400 positive (having that property) genes (ratio 45:1). However, I only have 6000 negative and 200 positive examples (ratio 30:1) for which this property has been confirmed (my training set).

I've read that if the sizes of the training sets do not reflect the prior odds, the FP an TP rate must be scaled in order to avoid over- or underestimating the false discovery rate.

My questions are:

  1. Do I have to scale/correct the FP and TP? If so, is this for the reason stated above (underestimation of the FDR) or some other reason?
  2. If scaling is necessary, how can I achieve it?
  3. Other performance measures, such as specificity and sensitivity, also use FP and TP (and TN&FN). Should I use corrected FP TP TN FN for these too?

A more general question:

I believe that what I refer to above as "estimated 'real' ratios (and sizes)" of the positive and negative sets are the prior odds. Is that correct?

I am not a statistician and I have a hard time trying to find good resources on the subject. This is mainly because false discovery rate seems to be primarily associated with Benjamini-Hochberg multiple testing correction. I'm not sure if the two applications of false discovery rate are in fact the same..

Any help would be greatly appreciated.

share|improve this question

1 Answer 1

False discovery is a concept due to Benjamini and Hochberg. It came about explicitly to as a criteria for adjusting p-values in multiple testing when a large number of tests are involved. This came up in the context of gene expression analysis using microarrays. The false discovery rate that you are using to evaluate your classifier is the same. There is a lot of literature on this in the medical journals where microarray analysis is often applied. The false discovery rate that you see in your data will be different from that in the population that you are interested in because of the difference in occurrences in the population compared to the training sample. So the adjustment you mention is appropriate to get an unbiased estiamte for the population. But this can only be done if you know the ratio of true negatives to true positives in the population which is what you claim you have here. But this would not be commonly known. In your case you would also have to adjust for other measures that compare TP to FP and TN to FN. This problem does not usually come up because the samples are usually taken as random samples from the target population. A more serious question is why you have this imbalance in the first place.

share|improve this answer
    
Thanks a lot for that Michael. I don't exactly know the ratio between true negatives and true positives in the population, but I can get a reasonable estimation based on some biological knowledge. Furthermore, I don't sample from the target population randomly because, again, we can get a much more reliable negative set for our purposes from the biology of the positive and negative examples. I might however be able to reduce the size of the negative set, so the ratios between the training and target populations correspond better.. Would that avoid the need to adjust the performance measures? –  RobinVdL Jun 18 '12 at 15:15
    
@RobinVdL I do not think there is ever a advantage to trhowing out data and thereby giving up some of the information in the data. –  Michael Chernick Jun 18 '12 at 15:31

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.