I'm having a problem to predict hits from original features. I tried using LDA on original matrix but the thing is that probability of getting a hit vs non-hit is 95% vs 5%. That said after running LDA I get true positive rate of 5% and true negative rate of 96%. Now, my understanding of LDA makes me think that when it separates hits from non-hits into two groups the later gets more attention, however what I really want is to minimize the false discovery rate, even better to control it. The question is what algorithm is best suited for this? I have two binary coexclusive groups (Hit/Not-hit).
A classical binary classifier that you could use is logistic regression, and for this particular case the Firth's method seems adequate to correct for the "rarity" of your hits.
See here for more info
You can use any binary classifier e.g. Logistic Regression. The problem is more tricky at model selection stage, more precisely - how are you going to select the model. Here you will need to estimate the performance (on some held-out set). Your case is typical for real-world data since you have big class imbalance. Computing performance indicator as accuracy will be very biased. I propose you to use a balanced measure of accuracy (see Talk Slides).
I think your problem is actually evaluating a model's output, not finding a good model per say. Unfortunately, that's the more difficult problem.
Look at ROC and AUC- a good source is http://www-bcf.usc.edu/~gareth/ISL/ The R package ROCR is quite useful.
Scoring Rules can be very helpful as well.