9
$\begingroup$

I am using a standard version of logistic regression to fit my input variables to binary output variables.

However in my problem, the negative outputs (0s) far outnumber the positive outputs (1s). The ratio is 20:1. So when I train a classifier, it seems that even features that strongly suggest the possibility of a positive output still have very low (highly negative) values for their corresponding parameters. It seems to me that this happens because there are just too many negative examples pulling the parameters in their direction.

So I am wondering if I can add weights (say using 20 instead of 1) for the positive examples. Is this likely to benefit at all? And if so, how should I add the weights (in the equations below).

The cost function looks like the following: $$J = (-1 / m) \cdot\sum_{i=1}^{m} y\cdot\log(h(x\cdot\theta)) + (1-y)(1 - \log(h(x\cdot\theta)))$$

The gradient of this cost function (wrt $\theta$) is:

$$\mathrm{grad} = ((h(x\cdot\theta) - y)' \cdot X)'$$

Here $m$ = number of test cases, $x$ = feature matrix, $y$ = output vector, $h$=sigmoid function, $\theta$ = parameters we are trying to learn.

Finally I run the gradient descent to find the lowest $J$ possible. The implementation seems to run correctly.

Improve this question
$\endgroup$
12
  • $\begingroup$ Hi, I have exactly the same problem that you described. In my data a lot of examples are negative and very few positive, and for me it's more important to correctly classify the positive, even if that means to miss-classify some negatives. It appears that I'm also applying the same methods as you were, since I'm using the same Cost Function and gradient equations. So far, I have run a few tests and I obtained the following results: - With 7 parameters, Training sample size: 225000, Test sample size: 75000 Results: 92% accuracy, although in the positives cases only 11% w $\endgroup$
    – Cartz
    Commented Feb 3, 2014 at 10:35
  • 1
    $\begingroup$ What you are doing is confusing a loss function with maximum likelihood. The unweighted mle is doing the "right thing" from an inferential perspective, and reflecting how rare the outcome is for each covariate specification. You could also have separation - this would happen that a particular set of covariates that can perfectly predict the response in the training data - this would lead to large negative values. $\endgroup$
    – probabilityislogic
    Commented Feb 3, 2014 at 11:34
  • 2
    $\begingroup$ Classification is not a good goal and is not the way logistic regression was developed. It is the notion of classification that causes all the problems listed here. Stick to predicted probabilities and proper accuracy scoring rules $\endgroup$
    – Frank Harrell
    Commented Feb 3, 2014 at 13:31
  • 1
    $\begingroup$ @arahant That's only partially true. A binary logistic regression with a logit link is still valid in that the coefficients on your covariates are MLE and reflect the effect those variables have on the odds of class 1 compared to class 0. However, in a case-control design, the intercept is always fixed to reflect the proportion of class 1 to class 0, and it is perfectly valid to adjust the intercept term to assign classes in line with, e.g., some cost function of misclassification, or some other process, because this doesn't change coefficients on variables. $\endgroup$
    – Sycorax ♦
    Commented Feb 3, 2014 at 18:07
  • 1
    $\begingroup$ Where did anyone get the idea that a cutoff is needed/wanted/desireable? $\endgroup$
    – Frank Harrell
    Commented Feb 3, 2014 at 20:29

2 Answers 2

Reset to default
8
$\begingroup$

That would no longer be maximum likelihood. Such an extreme distribution of $Y$ only presents problems if you are using a classifier, i.e., if you are computing the proportion classified correctly, an improper scoring rule. The probability estimates from standard maximum likelihood are valid. If the total number of "positives" is smaller than 15 times the number of candidate variables, penalized maximum likelihood estimation may be in order.

Improve this answer
$\endgroup$
5
  • $\begingroup$ Frank, is there a reference or something to support your "15 times..." detail? I have similar unbalance in some data that I am using logistic regression for in place of a ROC method some other researchers developed. I have recently come across the small-sample bias and added an option for Firth's bias-reduction as a fitting option in my code/package. As I'm writing this up for a journal it would be useful to have something to cite along side rules of thumb like this. Apologies if the reference is your RMS book as that is sat on my shelves, but haven't looked there yet. $\endgroup$
    – Gavin Simpson
    Commented Jul 24, 2013 at 15:53
  • $\begingroup$ There are papers on small-sample bias and the value of the Firth penalization. I don't have those handy. Regarding 15:1 see biostat.mc.vanderbilt.edu/wiki/pub/Main/FrankHarrell/… $\endgroup$
    – Frank Harrell
    Commented Jul 24, 2013 at 15:58
  • $\begingroup$ Thanks Frank - the 15:1 issues was what I was most after. I have some publications on the small-sample bias and Firth's method - but if you did have something to hand eventually I'd be most grateful if you let me know what it was. $\endgroup$
    – Gavin Simpson
    Commented Jul 24, 2013 at 16:03
  • 3
    $\begingroup$ Just in case anyone else should misread the above as I did at first. The 20:1 in the question is the ratio of negative to positive observations. The 15:1 in Frank Harrell's answer is something else: the ratio of positive observations to candidate independent variables. $\endgroup$
    – Adam Bailey
    Commented Jul 25, 2013 at 16:45
  • $\begingroup$ An extreme distribution also presents a problem by increasing the chance of quasi-complete separation, especially if you have categorical predictors. Penalisation helps here as well. $\endgroup$
    – probabilityislogic
    Commented Feb 3, 2014 at 13:35
3
$\begingroup$

In cases like this, it is often better to use a flexible link, instead of the logistic link, that can capture this asymmetry. For example a skew-normal, GEV, sinh-arcsinh, and the references therein. There are many others but I cannot post more than 2 links.

Improve this answer
$\endgroup$
1
  • 1
    $\begingroup$ Can you provide any explanation for other link functions are better? $\endgroup$
    – D.W.
    Commented Jul 22, 2016 at 0:32

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.