1
$\begingroup$

I'm having a problem to predict hits from original features. I tried using LDA on original matrix but the thing is that probability of getting a hit vs non-hit is 95% vs 5%. That said after running LDA I get true positive rate of 5% and true negative rate of 96%. Now, my understanding of LDA makes me think that when it separates hits from non-hits into two groups the later gets more attention, however what I really want is to minimize the false discovery rate, even better to control it. The question is what algorithm is best suited for this? I have two binary coexclusive groups (Hit/Not-hit).

$\endgroup$
  • 1
    $\begingroup$ What software are you using? In LDA you can adjust the penalty for being wrong in different ways, but the way to do this depends on software. $\endgroup$ – Peter Flom Nov 5 '12 at 15:17
  • $\begingroup$ Can you post more info about the data? There is a plethora of binary classifiers you can try, so some details are crucial to show you the right direction. $\endgroup$ – user88 Nov 6 '12 at 0:18
  • $\begingroup$ The software is R, MASS package. The data is different features extracted from images (e.g. Haralick texture features) minus reference value per each feature. The manual annotation was done by expert in biological field. He marked every image as class 1 if the image was different from reference image in his opinion. $\endgroup$ – Sergej Andrejev Nov 6 '12 at 15:24
  • $\begingroup$ By "true positive rate" and "true negative rate" I think you mean "predicted positive rate" and "predicted negative rate", otherwise this would be a very, very poor (or very lopsided) classifier... $\endgroup$ – shadowtalker Oct 2 '14 at 1:41
  • $\begingroup$ But this is a great question: I would love to know which classifier is the most robust to "rare" outcomes. $\endgroup$ – shadowtalker Oct 2 '14 at 1:44
1
$\begingroup$

A classical binary classifier that you could use is logistic regression, and for this particular case the Firth's method seems adequate to correct for the "rarity" of your hits.

See here for more info

$\endgroup$
  • $\begingroup$ Would you mind expanding on this a bit? I've never heard of this method, or for that matter of exact logistic regression. The page is all code and I'd like some of the theory. And the link on that page to the "nice summary" is broken. $\endgroup$ – shadowtalker Oct 2 '14 at 1:47
  • $\begingroup$ @ssdecontrol, I believe LionelB is referring to something similar to Scortchi's answer here: stats.stackexchange.com/questions/67903/… $\endgroup$ – Zhubarb Jan 12 '15 at 11:51
0
$\begingroup$

You can use any binary classifier e.g. Logistic Regression. The problem is more tricky at model selection stage, more precisely - how are you going to select the model. Here you will need to estimate the performance (on some held-out set). Your case is typical for real-world data since you have big class imbalance. Computing performance indicator as accuracy will be very biased. I propose you to use a balanced measure of accuracy (see Talk Slides).

$\endgroup$
0
$\begingroup$

I think your problem is actually evaluating a model's output, not finding a good model per say. Unfortunately, that's the more difficult problem.

Look at ROC and AUC- a good source is http://www-bcf.usc.edu/~gareth/ISL/ The R package ROCR is quite useful.

Scoring Rules can be very helpful as well.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.