2
$\begingroup$

While building a logistic regression model, I have imbalanced binary classes in 80:20 ratio.

  1. Does imbalance have any impact on the performance of a logistic model? If it depends on the severity, what's the criteria to consider a dataset as imbalanced?
  2. Does it have anything to do with the fact that we are predicting probabilities and imbalance might just skew the probability distribution and the inaccuracies due to it can be controlled by choosing an appropriate threshold for cut-off?
  3. Is there a function that can calculate the most optimum cut-off directly in R?

Thanks

$\endgroup$

1 Answer 1

2
$\begingroup$

The log-likelihood function of a binary classification model happends to be the same as KL divergence between your model and the real model. So if your model parameters are obtained by maximizing the log-likelihood (or minimizing the log-loss), the KL divergence will be minimized: $$ \begin{align}{LL}_{LR} & = \frac{1}{N} \log \prod_i \pi^{y_i}(1-\pi)^{1-y_i} \\&= \frac{\sum_i y_i}{N} \log \pi + \frac{\sum_i 1-y_i}{N}\log (1-\pi) \\ &= KL(p||q) \end{align} $$ Where $\frac{\sum_i y_i}{N}$ is the probability of observing $y_i$ in the "true model" $p$, $\pi$ is the probability of observing $y_i$ in the proposed model $q$.

In another word, the imbalance will not skew your estimation of the probability distribution.

But the number of samples will. If $\frac{\sum_i y_i}{N}$ can not capture well the proportion of positive samples in the real model, the so called "imbalance" may cause a porblem. For example if the positive rate of the population is 0.000001 (the "real model"), where you can easily get 100 samples with 0 positive case in it, so that your sample positive rate is $\frac{\sum_i y_i}{N}=0$, which is far from the real positive rate, and as a result your model will behave as if it's "skewed". But if you can increase your sample size to, say $1 \times 10^{20}$, then the sample postive rate will be much closer to the one of the population.

So to the question "how imbalance is my data?", the answers can be found by answering "how many samples do I need to make sure the sample positive rate can reflact the real positive rate?", or "With my current sample size $n$, what is the confidence interval of the sampled positive rate?". This are basic statistical problems where you can encounter a lot in hypothesis testing scenarios, I won't say much about it.

What if you find out that your sample size can not give you a significant estimation of the real positive rate, and you can't get more samples than you currently have? One way is to use resampling methods such as bootstrap to better capture the population distribution with limited sample size.

Another thing is, usually the model evaluation criteria is different from the model training criteria. For example your model may be trained by maximizing the log-likelihood, but it's evaluated by precision, recall, or f1-score. If you delibritely want to increase, for example recall rate, you should modify your machines understand of the "real model" by, say increasing the weights of the positive samples in the training set, and do a cross-validation on the weighting mechanisms to pick the best one.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.