I created a logistic regression model using PySpark ML. My feature set consists of both categorical and continuous features, and I ran the following to pre-process them:

  • Categorical features: All of them have less than 10 choices, and most of them are booleans. I did one-hot encoding on top of label encoding.
  • Numerical features: Some of them have very large range of domain, and some have outliers that are much larger than the majority. I stripped out these outliers by capping them at 2nd and 98th percentile and

    for ccol in columnsWithOutliers:
    cmin, cmax = df.approxQuantile(ccol, [0.02,0.98], 0.02)
    df = df.withColumn(ccol, when(col(ccol) > cmax, cmax).when(col(ccol) < cmin, cmin).otherwise(col(ccol)))

On top of this, I'm assembling everything as one vector and running MinMaxScaler to scale all features in range [0,1].

Unfortunately, my model hasn't been generalizing at all no matter what I do. I tried removing some features and did CrossValidation with parameter search, but the accuracy (and f-1 score) doesn't seem to improve much; my data is heavily skewed, in that only 20% of them have label "true" and 80% of them have label "false". But I think logistic regression should not be affected by data imbalance.

Right now, the model does almost perfect job classifying ones with "true" label, but only correctly classifies around 10% of samples with "false" label. Hence, although my accuracy/precision is around 80%, my model is performing horribly.

Any ideas on what I could do to increase overall performance of the model? I'm not willing to adjust the threshold since I need the probability instead of the predicted labels (I want to improve the model so that the probability properly lie between range of [0,1], with 0.5 as the borderline).

  • $\begingroup$ To see if the skewed data is causing the problem, I implemented stratified sampling instead of random sampling for train/test split. Although the result improved a bit, accuracy/precision for samples with label 0 is only around 25%. Any ideas would be greatly appreciated! $\endgroup$
    – Brian C
    Feb 21, 2019 at 1:02

1 Answer 1


Firstly, to know the learnability, train it on the validation set and see the score. This will set a theoretical Max you can achieve on the dataset.

In scikit learn you can put weights = 'balanced' so it takes class imbalance into account. Or use upsampling prior to CV. I think the weights option works best. This should fix your problem


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.