0
$\begingroup$

I have a bunch of data (over a million cases) with a few variables. One of these variables classifies the data into one of two groups: premature and mature. I want to develop some statistical technique/function so that when I am given new cases whose group is unknown (premature or mature), I can use it to classify them into one of the two groups. From my knowledge it would seem that discriminant analysis is the best technique or maybe logistic regression? However my data is really unbalanced (more mature than premature). So should I purposely make it a 50/50 amount for my training data set and then use the technique? And is there any other technique that is effective in achieving what I aim to do that you would recommend?

Thanks

$\endgroup$
1
$\begingroup$

Linear discriminant analysis is an option, but it is my understanding that it is not often used anymore because other methods have supplanted it (or at least that's what professors told me to justify never teaching me it).

What do you mean when you say "really unbalanced"? Class imbalance may be a problem, but many classification algorithms can handle it. If it is really bad (e.g., 98% mature), there are methods to address class imbalance, but it depends on what algorithm you choose. There's an entire tag here on CV for the issue.

Logistic regression is another option, and it can handle rare events. I suggest installing the caret package for R, as it has a large number of algorithms to choose from. Popular methods for classification—assuming you have continuous inputs—include nearest neighbors, logistic regression, decision trees, support vector machines, and neural networks. Each of these is worth trying and can be implemented rather easily in R via the caret and related packages.

You ask for the "best statistical method," which is a subjective question. Personally, I find decision trees to be particularly appealing because they are easy to interpret and can perform just as well as more complicated algorithms, especially if you are just classifying something in one of two classes.

Decision trees basically start with the whole dataset and then split it in two based on a value of some feature (i.e., independent variable, input variable), then it takes the two resulting groups and splits them in two, etc. The algorithm makes the splits where it can increase homogeneity in the two resulting groups. Eventually, it makes a prediction based on these splits. It will end up saying some rules for classification, like: "If their age was greater than 10 but less than 20, they were a female, and they had high math scores, then we categorize them as Class A."

I really like random forests, as they use bootstrap methods do a bunch of decision trees on a dataset, and then each tree kind of votes on how to average out predictions.

How about a real example?

Consider a few years back, when a bunch of fraternity brothers at the University of Oklahoma for singing a racist song on a bus. I have data from people six days after that event, where I asked them a binary question: Do you agree with the fraternity being shut down and two of the students being expelled? I also asked them a bunch of questions: Their gender, race, prejudice, political orientation, etc.

Let's try to make a model where we try to predict whether or not people agree with the punishment.

Whether or not they agreed is called Agree, and it is coded for 1 if they agree with the punishment (n = 112) and -1 if they did not like the fraternity brothers getting that punishment (n = 35). Here we can already see the class imbalance.

Let's predict it from anti-Black prejudice (B_prej), if they were White or not (white_filter, White coded 1, non-White coded 0), political identification as conservative/Republican (polid), and gender (gender, male coded 1, female coded 2). It is this simple in the randomForest package:

library(randomForest)
model <- randomForest(formula=Agree ~ B_prej + white_filter + polid + gender, 
                      data=data, na.action=na.omit)

Then you could use this model to classify future cases using the predict() function. Machine Learning with R is a gentle introduction to classification with good examples. The caret website is also great, which has a whole section on dealing with class imbalance.

Again, I said the best part about this was how easy it was to interpret these (compared to something like a neural network). We can extract how important each of these predictors was:

model$importance
             MeanDecreaseGini
B_prej              25.545972
white_filter         3.202190
polid                9.768993
gender               4.116992

Anti-Black prejudice was the most important feature to split on. Now, with a little help from our friends at CV, we can plot this to see what the tree looks like: enter image description here

Let's follow one path. If people were White (that is, got N for white_filter < 0.5) and high in political conservatism (that is, got N for polid < 5.75), then they were assigned the class -1 (regardless of gender), which means they disagreed with the fraternity kids getting punished for the racist song.

This is just one option in many options for classification, but I suggest random forests because they give a great balance of predictive accuracy and interpretability, especially if you are just categorizing into two categories.

$\endgroup$
2
  • $\begingroup$ Thank you so much! Incredibly helpful and easy to understand! Yes the data is really unbalanced like you said 99%. My skills are rather basic when it comes to R so I've been using SPSS modeler and did some decision trees (Random trees and C&R) like you suggested and they were fantastic! Very simple to follow and adequate predictive accuracy. Thanks a lot!! $\endgroup$ – Lucy May 28 '17 at 17:42
  • $\begingroup$ R has a tough learning curve, but I think it is worth it. That Machine Learning with R book I linked to in my answer makes it very user-friendly. RStudio is a very user-friendly IDE, as well. If you just Google "class imbalance in random forests," you will get many suggestions. This is both good and bad: It's good because people are there to help. It is bad because, as a professor of mine likes to say, if there are many ways to do something, then there is likely no one "best" way. $\endgroup$ – Mark White May 28 '17 at 18:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.