I created a logistic regression model using PySpark ML. My feature set consists of both categorical and continuous features, and I ran the following to pre-process them:
- Categorical features: All of them have less than 10 choices, and most of them are booleans. I did one-hot encoding on top of label encoding.
Numerical features: Some of them have very large range of domain, and some have outliers that are much larger than the majority. I stripped out these outliers by capping them at 2nd and 98th percentile and
for ccol in columnsWithOutliers: cmin, cmax = df.approxQuantile(ccol, [0.02,0.98], 0.02) df = df.withColumn(ccol, when(col(ccol) > cmax, cmax).when(col(ccol) < cmin, cmin).otherwise(col(ccol)))
On top of this, I'm assembling everything as one vector and running MinMaxScaler to scale all features in range [0,1].
Unfortunately, my model hasn't been generalizing at all no matter what I do. I tried removing some features and did CrossValidation with parameter search, but the accuracy (and f-1 score) doesn't seem to improve much; my data is heavily skewed, in that only 20% of them have label "true" and 80% of them have label "false". But I think logistic regression should not be affected by data imbalance.
Right now, the model does almost perfect job classifying ones with "true" label, but only correctly classifies around 10% of samples with "false" label. Hence, although my accuracy/precision is around 80%, my model is performing horribly.
Any ideas on what I could do to increase overall performance of the model? I'm not willing to adjust the threshold since I need the probability instead of the predicted labels (I want to improve the model so that the probability properly lie between range of [0,1], with 0.5 as the borderline).