Binary classification with imbalanced classes [duplicate]

Question

I have a manufacture dataset of 65 million rows corresponding to 65 millions distinct items.

Out of those 65 millions, I have 60,000 of them that failed a certain test, thus I have very imbalanced classes.

I have about 200 to 300 variables/features that describe them that are both categorical (2 levels or more) and numerical.

So

• y = failed items = 1 or 0 (binary)

• X = $X_1$ ... $X_{300}$ (categorical and numeric).

What is the proper methodology or model to use to root cause the important variables that contribute to the failed items? In other words, how to I identify the top variables that may be responsible for the failure?

Answer 1

Logistic regression would be an easy way of doing this. With 60'000 events out of 65'000'000 measurements, you have abundant data for the 300 predictors. However, datasets this large always make me wonder whether all units (the rows) are independent. Without more details about your data I dare not make assumptions about this.

Additionally, if you are interested in seeing which of your variables has the largest effect size, you might want to standardize the predictor variables and compare the effect per standard deviation.

Answer 2

Either way you should transform your categorical variables into dummy ones(and consider multicollinearity).

I agree with previous answer observation on possible non-independence between samples(items). If you have a reason to believe this is the case, many methods wont apply.

If you think that such issue does not exist i would go with a simple Chi-Squared test to examine each variable separately, and a logistic regression for cross variables effects.

Answer 3

You do not have to use all of the 65 million data points. You can take a sample, and apply a set of your favorite tools and then see if the model generalizes to dropped data. This increases your options for the method a lot.

MCA-to normalize the data. https://en.wikipedia.org/wiki/Multiple_correspondence_analysis It's like PCA, but with nominal variables.

Failing is seldom always the same. You may want to look if there is many clusters within those 60,000 failures.