# PySpark ML: What to do when a logistic regression model is not generalizing?

I created a logistic regression model using PySpark ML. My feature set consists of both categorical and continuous features, and I ran the following to pre-process them:

• Categorical features: All of them have less than 10 choices, and most of them are booleans. I did one-hot encoding on top of label encoding.
• Numerical features: Some of them have very large range of domain, and some have outliers that are much larger than the majority. I stripped out these outliers by capping them at 2nd and 98th percentile and

for ccol in columnsWithOutliers:
cmin, cmax = df.approxQuantile(ccol, [0.02,0.98], 0.02)
df = df.withColumn(ccol, when(col(ccol) > cmax, cmax).when(col(ccol) < cmin, cmin).otherwise(col(ccol)))


On top of this, I'm assembling everything as one vector and running MinMaxScaler to scale all features in range [0,1].

Unfortunately, my model hasn't been generalizing at all no matter what I do. I tried removing some features and did CrossValidation with parameter search, but the accuracy (and f-1 score) doesn't seem to improve much; my data is heavily skewed, in that only 20% of them have label "true" and 80% of them have label "false". But I think logistic regression should not be affected by data imbalance.

Right now, the model does almost perfect job classifying ones with "true" label, but only correctly classifies around 10% of samples with "false" label. Hence, although my accuracy/precision is around 80%, my model is performing horribly.

Any ideas on what I could do to increase overall performance of the model? I'm not willing to adjust the threshold since I need the probability instead of the predicted labels (I want to improve the model so that the probability properly lie between range of [0,1], with 0.5 as the borderline).

• To see if the skewed data is causing the problem, I implemented stratified sampling instead of random sampling for train/test split. Although the result improved a bit, accuracy/precision for samples with label 0 is only around 25%. Any ideas would be greatly appreciated! – Brian C Feb 21 '19 at 1:02