# Adding weights for highly skewed data sets in logistic regression

I am using a standard version of logistic regression to fit my input variables to binary output variables.

However in my problem, the negative outputs (0s) far outnumber the positive outputs (1s). The ratio is 20:1. So when I train a classifier, it seems that even features that strongly suggest the possibility of a positive output still have very low (highly negative) values for their corresponding parameters. It seems to me that this happens because there are just too many negative examples pulling the parameters in their direction.

So I am wondering if I can add weights (say using 20 instead of 1) for the positive examples. Is this likely to benefit at all? And if so, how should I add the weights (in the equations below).

The cost function looks like the following: $$J = (-1 / m) \cdot\sum_{i=1}^{m} y\cdot\log(h(x\cdot\theta)) + (1-y)(1 - \log(h(x\cdot\theta)))$$

The gradient of this cost function (wrt $\theta$) is:

$$\mathrm{grad} = ((h(x\cdot\theta) - y)' \cdot X)'$$

Here $m$ = number of test cases, $x$ = feature matrix, $y$ = output vector, $h$=sigmoid function, $\theta$ = parameters we are trying to learn.

Finally I run the gradient descent to find the lowest $J$ possible. The implementation seems to run correctly.

• Hi, I have exactly the same problem that you described. In my data a lot of examples are negative and very few positive, and for me it's more important to correctly classify the positive, even if that means to miss-classify some negatives. It appears that I'm also applying the same methods as you were, since I'm using the same Cost Function and gradient equations. So far, I have run a few tests and I obtained the following results: - With 7 parameters, Training sample size: 225000, Test sample size: 75000 Results: 92% accuracy, although in the positives cases only 11% w – Cartz Feb 3 '14 at 10:35
• What you are doing is confusing a loss function with maximum likelihood. The unweighted mle is doing the "right thing" from an inferential perspective, and reflecting how rare the outcome is for each covariate specification. You could also have separation - this would happen that a particular set of covariates that can perfectly predict the response in the training data - this would lead to large negative values. – probabilityislogic Feb 3 '14 at 11:34
• Classification is not a good goal and is not the way logistic regression was developed. It is the notion of classification that causes all the problems listed here. Stick to predicted probabilities and proper accuracy scoring rules – Frank Harrell Feb 3 '14 at 13:31
• @arahant That's only partially true. A binary logistic regression with a logit link is still valid in that the coefficients on your covariates are MLE and reflect the effect those variables have on the odds of class 1 compared to class 0. However, in a case-control design, the intercept is always fixed to reflect the proportion of class 1 to class 0, and it is perfectly valid to adjust the intercept term to assign classes in line with, e.g., some cost function of misclassification, or some other process, because this doesn't change coefficients on variables. – Reinstate Monica Feb 3 '14 at 18:07
• Where did anyone get the idea that a cutoff is needed/wanted/desireable? – Frank Harrell Feb 3 '14 at 20:29

That would no longer be maximum likelihood. Such an extreme distribution of $Y$ only presents problems if you are using a classifier, i.e., if you are computing the proportion classified correctly, an improper scoring rule. The probability estimates from standard maximum likelihood are valid. If the total number of "positives" is smaller than 15 times the number of candidate variables, penalized maximum likelihood estimation may be in order.