negatives using, e.g., linear discriminant analysis (LDA), and perform false discovery rate (FDR) control for
positives. Here, the FDR is defined as the estimated fraction of
negatives in accepted
I have a large number of training data, and trained the model using LDA, then calculated LDA scores for training data. FDR for each score threshold
t was then simply calculated by
p are number of
positives in training data set with LDA scores higher than
t, respectively. If I want to control the FDR at 0.01, the minimum score with
n/p <= 0.01 was then used as the threshold
t. For prediction, if a new sample has LDA score higher then
t, I will accept it as
positive. Since the training data is large, this FDR control procedure should be statistically valid. So, can I state like this: "For all unknown samples predicted as
positives, whose LDA scores are higher than
t, the FDR in predicted
positive set is still lower than 0.01"?
Under real situation, I have another large number of data, and predict them by LDA trained from above training data, the estimated FDR is unfortunately higher than 0.01. Does this violate the statement made above?