Improving accuracy of a binary classification when the target is unbalanced I am working on the BRFSS dataset with the goal of predicting Diabetes. The dataset has 500,000 rows and 405 columns. It is a 0/1 classification problem, the ratio of 0 to 1 is 90:10. I tried using decision trees, logistic regression an ensemble of decision trees and logistic regression and my misclassification rate is almost 14% in all of these methods. 


*

*What should I do to increase the accuracy? 


I saw an earlier post which says subsampling or assigning different weights helps. But I am not sure about the ratio. 


*

*What would be the best ratio to start off with? 

*I am working using SAS. Is there a way to do subsampling in SAS? 

*I am also interested in trying out the weighted approach. Is there a way to implement this in SAS?


EDIT (28 Apr 2011)
I tried subsampling and my misclassification rate goes up from 14% to 23%. The ratio I used was 50:50 for classes 0 and 1. The original ratio in the data was 90:10 and using the data as it is gave 14% error. So I believe subsampling doesn't work for my data. Would you suggest any other  way to improve accuracy?
 A: The problem is more with the choice of the accuracy scoring rule.  Make sure that the ultimate goal is classification as opposed to prediction.  The proportion classified correctly is a discontinuous improper scoring rule.  An improper scoring rule is one that is optimized by a bogus model.  With an improper scoring rule such things as addition of a highly important predictor making the model less accurate can happen.  The use of log likelihood (or deviance) or the Brier quadratic scoring rule will help.  The concordance index C (which happens to equal the ROC area, making ROCs appear more useful than they really are) is a useful measure of predictive discrimination once the model is finalized.
A: Regarding decision trees, I would suggest the following. Assume that you have 10 training examples from class $C_1$ and 90 training examples from class $C_2$. You can use an ensemble of $N$ decision trees, where each tree is trained on 10 examples from $C_1$ and 10 randomly chosen examples from $C_2$. The decision of the ensemble may be the majority vote. You can play with different $N$ to see how it works.
A: Based on the output you shared, Maximum # of branches from a node is set at 2.  It's possible that raising that limit would give you more options for branches, especially if SAS can take continuous variables and break them up into categories.  It's data dredgy, but that's the game we're in, and as long as you crossvalidate you're on solid moral ground :-)
A: If you are using tree-based methods, you can play around with the splitting criterion. For example, at each step, choose the split that gives the highest weighted accuracy (the average of the two classes' accuracies).
This can be used as the basis for a random forest too, which should give you a good classifier.
I once used a similar process to boost precision while sacrificing recall. It worked very well (better than thresholding the scores from the classification algorithm which were very noisy anyway).
