7
$\begingroup$

I am working on the BRFSS dataset with the goal of predicting Diabetes. The dataset has 500,000 rows and 405 columns. It is a 0/1 classification problem, the ratio of 0 to 1 is 90:10. I tried using decision trees, logistic regression an ensemble of decision trees and logistic regression and my misclassification rate is almost 14% in all of these methods.

  • What should I do to increase the accuracy?

I saw an earlier post which says subsampling or assigning different weights helps. But I am not sure about the ratio.

  • What would be the best ratio to start off with?
  • I am working using SAS. Is there a way to do subsampling in SAS?
  • I am also interested in trying out the weighted approach. Is there a way to implement this in SAS?

EDIT (28 Apr 2011)

I tried subsampling and my misclassification rate goes up from 14% to 23%. The ratio I used was 50:50 for classes 0 and 1. The original ratio in the data was 90:10 and using the data as it is gave 14% error. So I believe subsampling doesn't work for my data. Would you suggest any other way to improve accuracy?

Improve this question
$\endgroup$
5
  • 5
    $\begingroup$ I can obtain a misclassification rate of just 10% with these data, sight unseen, by always predicting 0! Surely any method you try will improve on this, right? Perhaps some more information about the quality of your results would help us understand the situation better. $\endgroup$
    – whuber ♦
    Commented Apr 27, 2011 at 18:57
  • $\begingroup$ @whuber: Yes, we can always predict 0 and get misclassification rate(MIR) of < 10%. But, the models that I am using make prediction by reducing error, they do not look at the proportion of the classes (i.e.) 90:10. The dataset I am working on is BRFSS 2009 dataset which is available here in csv format. The target variable is DIABETE2. stats.stackexchange.com/posts/10049/edit. I am not sure about what you mean on information about the quality of my results. $\endgroup$
    – user3897
    Commented Apr 27, 2011 at 21:08
  • 2
    $\begingroup$ By "quality" I meant more details about the misclassification rate. For instance, there are two kinds of misclassification: predicting the 0's as 1's and predicting the 1's as 0's. Also, how are you doing the classifying? With such an imbalance of 0's and 1's it's unlikely that setting the classification threshold to a logit of 0 is best: you will need a different threshold. $\endgroup$
    – whuber ♦
    Commented Apr 27, 2011 at 22:06
  • $\begingroup$ @whuber I have provided the confusion matrix here Could you suggest a way of improving the accuracy based on this? $\endgroup$
    – user3897
    Commented Apr 28, 2011 at 14:49
  • $\begingroup$ Have you considered using a negative threshold? $\endgroup$
    – whuber ♦
    Commented Apr 28, 2011 at 14:55

4 Answers 4

Reset to default
6
$\begingroup$

The problem is more with the choice of the accuracy scoring rule. Make sure that the ultimate goal is classification as opposed to prediction. The proportion classified correctly is a discontinuous improper scoring rule. An improper scoring rule is one that is optimized by a bogus model. With an improper scoring rule such things as addition of a highly important predictor making the model less accurate can happen. The use of log likelihood (or deviance) or the Brier quadratic scoring rule will help. The concordance index C (which happens to equal the ROC area, making ROCs appear more useful than they really are) is a useful measure of predictive discrimination once the model is finalized.

Improve this answer
$\endgroup$
2
1
$\begingroup$

Regarding decision trees, I would suggest the following. Assume that you have 10 training examples from class $C_1$ and 90 training examples from class $C_2$. You can use an ensemble of $N$ decision trees, where each tree is trained on 10 examples from $C_1$ and 10 randomly chosen examples from $C_2$. The decision of the ensemble may be the majority vote. You can play with different $N$ to see how it works.

Improve this answer
$\endgroup$
2
  • 1
    $\begingroup$ @User3897: Whatever you do, definitely save some of your half million cases as a holdover sample for crossvalidation...I'm perplexed by your inability to find a better-predicting decision tree. Have you fully explored SAS's options, e.g., on your inputted rules for node size requirements for when a parent node can be split, or when a child can be formed? For the statistical significance of such splits? For the program's ability to group the values of continuous predictors into effective subsets? $\endgroup$
    – rolando2
    Commented Apr 28, 2011 at 2:49
  • $\begingroup$ @rolando2 I have provided my SAS Enterprise Miner output here. This contains the tree settings that I used. Could you suggest what I need to improve upon. $\endgroup$
    – user3897
    Commented Apr 28, 2011 at 15:10
0
$\begingroup$

Based on the output you shared, Maximum # of branches from a node is set at 2. It's possible that raising that limit would give you more options for branches, especially if SAS can take continuous variables and break them up into categories. It's data dredgy, but that's the game we're in, and as long as you crossvalidate you're on solid moral ground :-)

Improve this answer
$\endgroup$
2
  • 2
    $\begingroup$ Yes, rigorous cross-validation or bootstrapping will tell you how arbitrary the intervals are. It is not reasonable to expect the world to operate discontinuously however. Smooth fits are undoubtedly more realistic and will predict better at the margins. I'm not clear on why tree-based methods are as popular as they are. With absolutely huge databases the issues are less important. $\endgroup$
    – Frank Harrell
    Commented May 3, 2011 at 15:38
  • 1
    $\begingroup$ What I meant in the last sentence is that tree methods suffer less in terms of predictive accuracy when the sample size is exceedingly large. Methods based on a single tree can not improve upon continuous modeling for any sample size, however. And binning the data often results in gaming of the system at the margins. $\endgroup$
    – Frank Harrell
    Commented May 3, 2011 at 19:36
0
$\begingroup$

If you are using tree-based methods, you can play around with the splitting criterion. For example, at each step, choose the split that gives the highest weighted accuracy (the average of the two classes' accuracies).

This can be used as the basis for a random forest too, which should give you a good classifier.

I once used a similar process to boost precision while sacrificing recall. It worked very well (better than thresholding the scores from the classification algorithm which were very noisy anyway).

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.