10
$\begingroup$

I am building a propensity model using logistic regression for a utility client. My concern is that out of the total sample my 'bad' accounts are just 5%, and the rest are all good. I am predicting 'bad'.

  • Will the result be biassed?
  • What is optimal 'bad to good proportion' to build a good model?
Improve this question
$\endgroup$
1
  • $\begingroup$ I think its always better to have 50% of goods and 50% bads as per the rule of thumb. The out of og the model must not be biased on the sample. $\endgroup$
    – user40017
    Feb 11, 2014 at 11:50

5 Answers 5

Reset to default
7
$\begingroup$

I disagreed with the other answers in the comments, so it's only fair I give my own. Let $Y$ be the response (good/bad accounts), and $X$ be the covariates.

For logistic regression, the model is the following:

$\log\left(\frac{p(Y=1|X=x)}{p(Y=0|X=x)}\right)= \alpha + \sum_{i=1}^k x_i \beta_i $

Think about how the data might be collected:

  • You could select the observations randomly from some hypothetical "population"
  • You could select the data based on $X$, and see what values of $Y$ occur.

Both of these are okay for the above model, as you are only modelling the distribution of $Y|X$. These would be called a prospective study.

Alternatively:

  • You could select the observations based on $Y$ (say 100 of each), and see the relative prevalence of $X$ (i.e. you are stratifying on $Y$). This is called a retrospective or case-control study.

(You could also select the data based on $Y$ and certain variables of $X$: this would be a stratified case-control study, and is much more complicated to work with, so I won't go into it here).

There is a nice result from epidemiology (see Prentice and Pyke (1979)) that for a case-control study, the maximum likelihood estimates for $\beta$ can be found by logistic regression, that is using the prospective model for retrospective data.

So how is this relevant to your problem?

Well, it means that if you are able to collect more data, you could just look at the bad accounts and still use logistic regression to estimate the $\beta_i$'s (but you would need to adjust the $\alpha$ to account for the over-representation). Say it cost $1 for each extra account, then this might be more cost effective then simply looking at all accounts.

But on the other hand, if you already have ALL possible data, there is no point to stratifying: you would simply be throwing away data (giving worse estimates), and then be left with the problem of trying to estimate $\alpha$.

Improve this answer
$\endgroup$
8
  • $\begingroup$ This is only true though if you have enough data to adequately represent the minority class (which is usually the problem where there is a large class imbalance - the problem is the imbalance per se, but that you don't have enough samples from the minority class). In that case cross-validation based differential weighting of the positive and negative classes, adjustment of alpha, or multiplicative adjustment (all equivalent IIRC) would be a good idea to boost the minority class. Stratifying the data would be a cheap and cheerful equivalent. $\endgroup$
    – Dikran Marsupial
    May 6, 2011 at 10:21
  • $\begingroup$ @Dikran: I don't see how any of this fails to be true in that case: you don't have to represent the minority class, you just need to represent the relative difference to the majority. If you don't have enough data for that, then throwing away observations from the majority isn't going to help. $\endgroup$
    – Simon Byrne
    May 6, 2011 at 10:29
  • $\begingroup$ @Simon, I have agreed with you that stratification isn't a good idea unless you don't want to adjust alpha, reweight the patterns or adjust the output of the model. As I said the problem with class imbalance is not the imbalance per se, but that there is insufficient data for the minority class to adequately define the "relative difference to the majority". When that happens, on average it biases the output towards the majority class, and hence doing something to compensate for that bias is helpful. Stratification is one way of doing that, but not nearly the best. $\endgroup$
    – Dikran Marsupial
    May 6, 2011 at 10:38
  • $\begingroup$ @Simon, BTW thanks for the Prentice and Pyke reference, it looks useful. $\endgroup$
    – Dikran Marsupial
    May 6, 2011 at 10:39
  • 1
    $\begingroup$ @Dikran: It may well be biased for other models, but NOT for logistic regression, that's the whole point. $\endgroup$
    – Simon Byrne
    May 6, 2011 at 10:47
3
$\begingroup$

Asymptotically, the ratio of positive to negative patterns is essentially irrelevant. The problem arises principally when you have too few samples of the minority class to adequately describe its statistical distribution. Making the dataset larger generally solves the problem (where that is possible).

If this is not possible, the best thing to do is to re-sample the data to get a balanced dataset, and then apply a multiplicative adjustment to the output of the classifier to compensate for the difference between training set and operational relative class frequencies. While you can calculate the (asymptotically) optimal adjustment factor, in practice it is best to tune the adjustment using cross-validation (as we are dealing with a finite practical case rather than an asymptotic one).

In this sort of situation, I often use a committee of models, where each is trained on all of the minority patterns and a different random sample of the majority patterns of the same size as the minority patterns. This guards against bad luck in the selection of a single subset of the majority patterns.

Improve this answer
$\endgroup$
10
  • 2
    $\begingroup$ But is this pertinent to logistic regression? We don't need to describe the statistical distribution of either class, just the relative odds ratios (see my comment to @Henry). $\endgroup$
    – Simon Byrne
    May 5, 2011 at 15:50
  • $\begingroup$ In a univariate example perhaps, but if you have more than one explanatory variable then you need information about the distribution of patterns to correctly orient the "discriminant". $\endgroup$
    – Dikran Marsupial
    May 5, 2011 at 16:02
  • $\begingroup$ Wouldn't it be better to apply the multiplicative adjustment in log odds space before the logistic function is applied? $\endgroup$
    – rm999
    May 5, 2011 at 17:10
  • $\begingroup$ IIRC, the assymptitically optimal adjustment is to multiply by the ratio of operational to training set class frequencies, this is based on Bayes rule, so it is applied to the probabilities rather than the log-odds ratio. However, as we are just trying to correct for a mathematically intractable defficiency in the estimation, it probably doesn't matter too much how the adjustment is made, it is really just a "fudge-factor". $\endgroup$
    – Dikran Marsupial
    May 5, 2011 at 17:20
  • $\begingroup$ @Dikran: I don't understand what you mean about correctly orienting the "discriminant". Even in the multivariate case, logistic regression is still just computing relative odds ratios. $\endgroup$
    – Simon Byrne
    May 6, 2011 at 9:21
1
$\begingroup$

In theory, you will be able to discriminate better if the proportions of "good" and "bad" are roughly similar in size. You might be able to move towards this by stratified sampling, oversampling bad cases and then reweighting to return to the true proportions later.

This carries some risks. In particular your model is likely to be labelling individuals as "potentially bad" - presumably those who may not pay their utility bills when due. It is important that the impact of errors when doing this are properly recognised: in particular how many "good customers" will be labelled "potentially bad" by the model, and you are less likely to get the reweighting wrong if you have not distorted your model by stratified sampling.

Improve this answer
$\endgroup$
5
  • $\begingroup$ Actually, I don't think this is true for logistic regression: the odds-ratio parameter (which performs the discrimination) is invariant to stratification on the response variable. This is why it can be used for case-control studies. $\endgroup$
    – Simon Byrne
    May 5, 2011 at 15:05
  • $\begingroup$ @Simon: I don't disagree with your comment on the odds ratio, but I have seen people fail to take this back to consequences for the population correctly after stratified sampling when they had done so in other cases. For example, if you find that people for which factor A is true have twice the odds of being "bad" as those without factor A, this should not change with stratified sampling, but if you want to know what proportion of the population will be unnecessarily affected if you target those with factor A, then you need to carefully weight the information from your samples. $\endgroup$
    – Henry
    May 5, 2011 at 16:22
  • $\begingroup$ sorry, that wasn't the bit I disagreed with. It was the first bit: a consequence of the invariance is that once you have the data, there is no point in stratifying, you're simply throwing away data. (the story is different when it comes to the cost of collecting data, hence the existence of case-control studies). $\endgroup$
    – Simon Byrne
    May 5, 2011 at 17:04
  • $\begingroup$ @Simon: When you say "case-control study", do you mean that that you originally plan to take a sample of "bad" cases and a sample of "good" cases? With a higher proportion of "bad" cases than the small proportion in the population? If so, that is what I intended by "stratified sampling, oversampling bad cases" in my answer. $\endgroup$
    – Henry
    May 5, 2011 at 18:51
  • $\begingroup$ Yes, that is precisely what I meant as well. The question seemed to indicate that they already have data, hence there would be no point in stratifying. $\endgroup$
    – Simon Byrne
    May 6, 2011 at 9:18
0
$\begingroup$

There are many ways in which you can think of logistic regressions. My favorite way is to think that your response variable, $y_i$, follows a Bernoulli distribution with probability $p_i$. An $p_i$, in turn, is a function of some predictors. More formally:

$$y_i \sim \text{Bernoulli}(p_i)$$ $$p_i = \text{logit}^{-1}(a + b_1x_1 + ... +b_nx_n)$$ where $\text{logit}^{-1} = \frac{\exp(X)}{1+\exp(x)}$

Now does it matter it you have low proportion of failures (bad accounts)? Not really, as long as your sample data is balanced, as some people already pointed. However, if your data is not balanced, then getting more data may be almost useless if there is some selection effects you are not taking into account. In this case, you should use matching, but the lack of balance may turn matching pretty useless. Another strategy is trying to find a natural experiment, so you can use instrumental variable or regression disconinuity design.

Last, but not least, if you have a balanced sample or there is no selection bias, you may be worried with the fact the bad account is rare. I don't think 5% is rare, but just in case, take a look at the paper by Gary King about running a rare event logistic. In the Zelig package,in R, you can run a rare event logistic.

Improve this answer
$\endgroup$
-1
$\begingroup$

Okay so I work in Fraud Detection so this sort of problem is not new to me. I think the machine learning community has quite a bit to say about unbalanced data (as in classes are unbalanced). So there are a couple of dead easy strategies that I think have already been mentioned, and a couple of neat ideas, and some way out there. I'm not even going to pretend to know what this means for the asymptotics for your problem, but it always seems to give me reasonable results in logistic regression. There may be a paper in there somewhere, not sure though.

Here are your options as I see it:

  1. Oversample the minority class. This amounts to sampling the minority class with replacement until you have the same number of observations as the majority class. There are fancy ways to do this so that you do things like jittering the observation values, so that you have values close to the original but aren't perfect copies, etc.
  2. Undersample, this is where you take a subsample of the majority class. Again fancy ways to do this so that you are removing majority samples that are the closest to the minority samples, using nearest neighbor algorithms and so forth.
  3. Reweight the classes. For logistic regression this is what I do. Essentially, you are changing the loss function to penalize a misclassified minority case much more heavily than a misclassified majority class. But then again you are technically not doing maximum likelihood.
  4. Simulate data. Lot's of neat ideas that I've played with here. You can use SMOTE to generate data, Generative Adversarial Networks, Autoencoders using the generative portion, kernel density estimators to draw new samples.

At any rate, I've used all of these methods, but I find the simplest is to just reweight the problem for logistic regression anyway. One thing you can do to gut check your model though is to take:

-Intercept/beta

That should be the decision boundary (50% probability of being in either class) on a given variable ceteris paribus. If it doesn't make sense, e.g. the decision boundary is a negative number on a variable that is strictly positive, then you've got bias in your logistic regression that needs to be corrected.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.