logistic regression: the relation between sample proportion and prediction?

Suppose we have an dependent variable of 4 categories. The proportion of each of the four categories in the population is 4/10, 3/10, 2/10, and 1/10. The independent variables that we use are all continuous. I ran a multinomial logistic regression on a sample that reflect the relative porportion in the population. My result is not good and the misclassification rate is high.

My colleague thinks that it is because I did not "standardize" or equalize the porportion, so that the first group is overpowering the others, causing bias in the estimation.

My colleague's approach is to run three separate logistic regressions, A vs. Not A, B vs. Not B, C vs. Not C, D vs. Not D, and compare the estimated probability and pick the one with the highest probability. He thinks that it is important to "standardize" the good rate (defined as an event happening, so in the first model would be the rate of A, in the second the rate of B, etc) across the training set of the three models so that the probabilities are comparable across the three different model predictions. I have difficulty understand how the probabilities are comparable across.

So my questions are

1. Do we need to "standardize" or equalize the proportion for logistic, more specifically, multinomial logistic regression?
2. Can you compare logistic regression model predictions across different logistic regression?

Thank you!

(Edit to clarify what "standardize" mean here.)

• If you estimate a logistic regression on a sample then the average of the predicted probabilities for the sample will equal de proprotion of successes in the sample, this follows from the maximum likelihood equations (if you have a constant term). So your colleague is right. The analogum is found in linear regression: the regression line passes through the sample average (this follows from the OLS equations).
– user83346
Sep 11 '15 at 6:40

1.

Yes, the proportions do matter. In logistic regression, the estimated coefficients are log-odds, and when you exponentiate the beta coefficients of the independent variables you get the odds ratio, not the relative risk or something similar. Odds ratio, by definition, is similar to relative risk when the event is uncommon but as the event becomes more common, the odds ratio will be greatly distorted compared to the relative risk. I'll give you two examples, and for each example I'll first calculate the odds ratio and relative risk manually, then estimate the odds ratio via logistic regression:

In the first example we compare the association between an uncommon disease (0/1) and sex (0 = female, 1 = male). We'll assume that the disease is twice as common in males than in females. This gives a 2x2 cross table with the following cell counts:

a. Diseased males: 100 b. Diseased females: 50 c. Healthy males: 1900 d. Healthy females: 1950

The relative risk is (a/(a+c)) / (b/(b+d)) = 2. It is twice as common for men to have the disease.

The odds ratio is (a/c)/(c/d) = 2.05. Quite close to the relative risk. Now a logistic regression model gives:

disease <- c(rep(1, 150), rep(0,3850))
sex <- c(rep(1,100), rep(0,50), rep(1,1900), rep(0,1950))
summary(glm(disease ~ sex, family=binomial))

Coefficients:
Estimate Std. Error z value Pr(>|z|)
(Intercept)  -3.6636     0.1432 -25.580  < 2e-16 ***
sex           0.7191     0.1762   4.082 4.47e-05 ***


The estimated beta coefficient for sex is 0.7191. When we exponentiate to get the odds ratio we get exp(0.7191) = 2.05.

Ok, so no we'll try with another disease condition that is more common, and still more common in males. We'll assume that 30 percent of females and 60 males have the disease, just to make the results extreme:

a. Diseased males: 1200 b. Diseased females: 600 c. Healthy males: 800 d. Healthy females: 1400

The relative risk is (a/(a+c)) / (b/(b+d)) = 2. So the relative risk is unchanged though this disease is far more common.

The odds ratio is (a/c)/(c/d) = 3.5. So now the odds ratio is clearly distorted, almost twice the relative risk!

Running a logistic regression model (in R) confirms this:

disease <- c(rep(1, 1800), rep(0,2200))
sex <- c(rep(1,1200), rep(0,600), rep(1,800), rep(0,1400))
summary(glm(disease ~ sex, family=binomial))

Coefficients:
Estimate Std. Error z value Pr(>|z|)
(Intercept) -0.84730    0.04879  -17.36   <2e-16 ***
sex          1.25276    0.06682   18.75   <2e-16 ***


exp(1.25276) = 3.5

So yes, the proportions do matter because the (log) odds are estimated, not probability. The relationship between probability and odds is odds = p(1-p), and p = odds(1+odds).

2.

I don't understand the idea to "standardize the proportions". The approach suggest by your colleague may or may not be appropriate, depending on what it is that you want to study. I would like to know more about the dependent variable and independent variables of interest etc.

• The average of the predicted probabilities for the values in the sample are equal to the proportion of successes in the sample. (see my comment below the question) therefore, when the fraction of successes in the sample is very different from the fraction in the population then you will have to 'calibrate' or 'standardise' as the colleague says.
– user83346
Sep 11 '15 at 7:43
• I'm sorry, but I stil don't understand how the calibration or standardization is done. Say that a certain disease affects 1% of the population at any given time, and in our data, we have a prevalence of 5%. How can we standardize this and why?
– JonB
Sep 11 '15 at 7:49
• see pdf below, on page 10 below the table: this holds for the sample that you used to estiimate the coefficients, it does not hold 'out-of-sample', so if the fraction of successes in the sample differs strongly from the fraction in the population then you can have a problem. media.hsph.edu.vn/sites/default/files/…
– user83346
Sep 11 '15 at 8:08
• @JonasBerge @f coppens Thank you both! Let's say that the proportion in the population is 4/10, 3/10, 2/10, 1/10. Do you suggest that we should somehow "standardize" the proportion like my colleague does to 1/5 each? Thanks! Sep 11 '15 at 13:01
• @user: I still don't get it with the standardization so perhaps mr coppens can advise you in that. :)
– JonB
Sep 11 '15 at 13:03