I classified positives from negatives using, e.g., linear discriminant analysis (LDA), and perform false discovery rate (FDR) control for positives. Here, the FDR is defined as the estimated fraction of negatives in accepted positives.

I have a large number of training data, and trained the model using LDA, then calculated LDA scores for training data. FDR for each score threshold t was then simply calculated by n/p, where n and p are number of negatives and positives in training data set with LDA scores higher than t, respectively. If I want to control the FDR at 0.01, the minimum score with n/p <= 0.01 was then used as the threshold t. For prediction, if a new sample has LDA score higher then t, I will accept it as positive. Since the training data is large, this FDR control procedure should be statistically valid. So, can I state like this: "For all unknown samples predicted as positives, whose LDA scores are higher than t, the FDR in predicted positive set is still lower than 0.01"?

Under real situation, I have another large number of data, and predict them by LDA trained from above training data, the estimated FDR is unfortunately higher than 0.01. Does this violate the statement made above?

  • $\begingroup$ If you mean False Discovery Rate (FDR) than I don't think it makes sense to use it in this context. This is just not the right application for it. Why won't you just use ROC curve and find the right threshold to use from it? $\endgroup$ – itdxer May 29 at 9:30
  • $\begingroup$ Thanks. The FDR is commonly used in this area, so it is required. Is there a way to control the FDR in prediction? Since these data are unknown. $\endgroup$ – Elkan May 29 at 9:43
  • $\begingroup$ Can you provide a reference to the resource where it's being used in a very similar context? $\endgroup$ – itdxer May 29 at 11:42
  • $\begingroup$ This is my assumption. Currently no paper talks about this, neither in statistical papers. This is why I ask here for better suggestion. $\endgroup$ – Elkan May 29 at 12:53

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.