Tell me more ×
Cross Validated is a question and answer site for statisticians, data analysts, data miners and data visualization experts. It's 100% free, no registration required.
up vote 3 down vote favorite
3

I am trying to identify approximate 3% of the population for some characteristic feature. Standard decision tree or logistic regression gives too many false positives. Is there a chance that rules based classifier can improve performance? I would like to get approx 75% of recall with 95% of precission (i.e. FalsePositives <= 5% of Positives)

share|improve this question
2  
You should keep in mind that achieving what you want might be impossible. – Aniko Sep 7 '10 at 15:33
1  
'You should keep in mind that achieving what you want might be impossible.' I will stick this quote on the door of my office. – user603 Sep 7 '10 at 16:21
1  
You can add the output of a manually crafted rule based classifier as a predictor and see if the decision tree can improve upon it. – Iterator Aug 26 '11 at 18:48

3 Answers

up vote 4 down vote

Probably you just have unbalanced classes (3% to 97%, if I understood well) -- try balancing them (get this 3% of true ones and about equal number of false ones) and check the classifier build on this case. If you are worried that you have thrown out most of your data, iterate it few times and connect them with some simple blender, like voting. (More complex blenders will also suffer from unbalanced classes). You should also check some better classifiers than a single tree or logistic regression -- like SVM or Random Forest.
Of course you can also use some classifier immune to imbalance problem, like kNN or as you say some rule-based approach.

share|improve this answer
up vote 4 down vote

This is only valid for the logit: you can use another link function (complementary log-log or cloglog in short). This is a variation of the classical logit function that allows for assymetry (when one tail of the link function does not go to 0 at the same speed as the other tail goes to 1). I had a very good experience fitting one of these to a database with about 1% of 'ones'. this is a good starting point reference:

http://www.springerlink.com/content/cfakldm2renaj34m/

These can be fitted using the zelig package in R.

edit: there is a good comparaison with a logit link here: http://rss.acs.unt.edu/Rdoc/library/VGAM/html/cloglog.html

share|improve this answer
Hi Kwak, The Huettmann paper is interesting but I have a few questions. 1) Under Assessing the link Functions, what is meant by the "mean coefficients derived from the 1000 comparisons" (p44)? 2) What do the "Predictions" in Table2 refer to? Predicted probability or predicted number of nesting sites, or what? 3) The marbled murrelet data are claimed to be public domain. Do you know the source? Thanks. – Thylacoleo Sep 8 '10 at 3:44
I do not follow. The OP asked about decision trees and manually crafted rules, not GLMs. – Iterator Aug 26 '11 at 18:47
up vote 2 down vote

One strategy would be to use margin based methods with uneven margins (see this paper). Or you can use active learning to provide the learner with more balanced classes. Besides this, actually there are a number of other ways too that you can use to deal with imbalanced dataset. See this survey paper which discusses a number of techniques such as resampling, cost-sensitive approaches, active learning, etc (and evaluation methods).

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.