I have been working on a machine learning model using Spark (binomial) LogisticRegression. The dataset has what I think is a high degree of imbalance - roughly 1% of rows are labelled as events. The original author has used a weightCol to try and improve the model. The weights used are the inverse of the prevalence of the label (so those labelled as events are given a weight of 0.99, those non-events are weighted 0.01). A side effect of this is that the predictions from the model are skewed. Without using the weighting column, the mean of the probabilities output for the test set is roughly 1%; but with the weighting column, the average probability is around 30%. Is this a even valid use of the weightCol?
I have been reading lots of other articles/questions and I'm not convinced either way...I gather that this concept of weighting might have come from King and Zheng 2001 which makes it sound like it's meant for correcting under/oversampling (e.g. my data set has 1% events but I know the real rate is 5%).
Is there a way to correct the skewed probabilities from using weighting in the manner I described? I only arrived at the paper I cited from trying to answer this very question; in the paper they propose that only the intercept needs adjusting and give a formula for doing so (one that my Stack Exchange skills - or lack thereof - prohibit me from listing, but you can see it in the first answer here). If the formula is even applicable in our case, for the weighting I described, the sample is weighted as 1 minus the actual weight anyway (this makes the log term evaluate to ln(1), which is 0, and so the intercept is unchanged!).
I was wondering if, with the weighting that has been applied, we have effectively evened out the samples and that the y-hat value I need to plug into the formula is 0.5 (like in the link, actually). But then it just feels like I'm making up my own values...hence my cry for help!
I think I understand how I could use that formula to correct for undersampling (by undersampling I mean using fewer non-events to make the events seem more prevalent), but I am less keen on this approach as when I tried it, the precision was adeversely affected (which makes sense to me as the model has been trained on fewer non-events).
I have been asking myself why people would implement ways to do under/oversampling and weighting in Spark, which both skew the probabilities, without implementing a way to correct the probabilities (i.e. adjust the intercept). Maybe it's that they're both meant to be used together, and that weightCol is actually to correct for under/oversampling anyway?
