5
$\begingroup$

I have fit two logistic regression models in which the probability for a student to succeed a course is fit against how many hours he/she studied, where the students of group one used one type of course and the other group used another type of course.

I now am reasoning about any significant difference between the two models, was one type of course overall better at getting its students to succeed than the other one? I plotted the logistic regression model and I notice the two curves (one for each type of course) run almost in parallel, almost having the same slope at any point on the two curves. However, one curve lays higher than the other one at all times, meaning the probability of succeeding is higher for any hours of study using this type of course. Can I now conclude this type of course is overall better?

$\endgroup$
3
  • $\begingroup$ Parallel lines mean that the relationship between hours studied and the probability of succeeding is the same in the two courses (i.e. an hour more study increases the odds of success similarly for both courses). Lines that have different intercepts (i.e. one is higher than the other) means that the general probabiltiy of succeeding is higher in one course compared to the other. You could quantify that using estimated marginal means. $\endgroup$ Commented Dec 22, 2019 at 14:16
  • $\begingroup$ Is there a way to statistically test this difference? I assume I can already hint at the conclusions you stated in your reply, but to really draw any concrete conclusions I would have to perform a test of some kind to support it. I'm using R if maybe you have an easy example/function I should look into. $\endgroup$
    – Astarno
    Commented Dec 22, 2019 at 14:21
  • $\begingroup$ I read your response here as well: stats.stackexchange.com/questions/316801/… and did the testing for the slopes already. Maybe you could ellaborate on that response to say where I would read off the required information about the intercept or what would have to change about that test if that makes it easier. $\endgroup$
    – Astarno
    Commented Dec 22, 2019 at 14:22

1 Answer 1

5
$\begingroup$

Let's look at three possible models:

\begin{align} \mathrm{logit}(y) &= \beta_0 + \beta_1\cdot \mathrm{hours} + \beta_2\cdot \mathrm{course} + \beta_{3}\cdot \mathrm{hours}\times\mathrm{course}\tag{1} \\ \mathrm{logit}(y) &= \beta_0 + \beta_1\cdot \mathrm{hours} + \beta_2\cdot \mathrm{course}\tag{2}\\ \mathrm{logit}(y) &= \beta_0 + \beta_1\cdot \mathrm{hours} \tag{3} \end{align} where $\mathrm{hours}$ is a continuous variable and $\mathrm{course}$ is a dummy variable for one of the two courses. Model $(1)$ allows for the relationship between hours and probability of success to have i) different intercepts and ii) different slopes (i.e. non-parallel slopes). Model $(2)$ allow for the relationship between hours and probability of success to have different intercepts. Model $(3)$ assumes that the relationship between hours and probability of success is identical for both courses.

We can use a likelihood ratio test to check each simpler model agains the more complex one. Here is an example using R with simulated data. The simulation assumes that $\beta_{3} = 0$, i.e. that the lines are parallel. Specifically, I used model $(2)$ with $\beta_{0} = -65/7$, $\beta_{1} = 5/21$ and $\beta_{3} = 9/10$:

n <- 1000

set.seed(142857)

hours <- runif(n, 1, 60)
course <- rbinom(n, 1, 0.5)

beta0 <- -(65/7)
beta1 <- (5/21)
beta2 <- 0.9

z <- beta0  + beta1*hours + beta2*course 
pr <- 1/(1 + exp(-z))
y <- rbinom(n, 1, pr)

dat <- data.frame(
  y = y
  , hours = hours
  , course = factor(course, levels = c(0, 1), labels = c("course1", "course2"))
)


mod_3 <- glm(y~hours*course, family = binomial, data = dat)
mod_2 <- glm(y~hours + course, family = binomial, data = dat)
mod_1 <- glm(y~hours, family = binomial, data = dat)

# Likelihood ratio tests
anova(mod_2, mod_3, test = "LRT") # Test model 1 vs model 2
anova(mod_1, mod_2, test = "LRT") # Test model 2 vs model 3

The likelihood ratio test comparing model $(1)$ vs. $(2)$ is:

Model 1: y ~ hours + course
Model 2: y ~ hours * course
  Resid. Df Resid. Dev Df Deviance Pr(>Chi)
1       997     442.24                     
2       996     440.05  1   2.1889    0.139

As expected, there is little evidence suggesting that the model containing an interaction (i.e. non-parallel lines) is warranted. Now let's compare model $(2)$ with $(3)$:

Model 1: y ~ hours
Model 2: y ~ hours + course
  Resid. Df Resid. Dev Df Deviance  Pr(>Chi)    
1       998     458.20                          
2       997     442.24  1   15.959 6.474e-05 ***

There is a lot of evidence suggesting that different intercepts are warranted (i.e. parallel slopes but one is higher than the other). Hence, we would assume that the overall probability of success is different for the two courses. This is of course expected, because we simulated it with an odds ratio of $2.46$. The odds ratio in the simulated data is:

summary(mod_2)

Coefficients:
              Estimate Std. Error z value Pr(>|z|)    
(Intercept)    -9.6567     0.6842 -14.114  < 2e-16 ***
hours           0.2525     0.0174  14.507  < 2e-16 ***
coursecourse2   0.9837     0.2531   3.886 0.000102 ***

Hence, the estimated odds ratio for course 2 vs. course 1 is $\exp(0.9837) = 2.67$. We would interpret this as follows: "The odds of succeeding for participants using course 2 were 2.67 times higher than the odds of participants using course 1".

To make it even clearer, we could calculate and compare the average probability of succeeding for both course types using emmeans:

library(emmeans)
emmeans(mod_2, "course", transform = "response")

 course   prob     SE  df asymp.LCL asymp.UCL
 course1 0.121 0.0231 Inf    0.0754     0.166
 course2 0.268 0.0360 Inf    0.1979     0.339

Confidence level used: 0.95

On average, the probability of success is $0.121$ for participants of course 1 and $0.268$ for participants of course 2. This corresponds to an odds ratio of $2.67$.

Here is a plot of the three models on the probability scale:

Prob_scale

An on the log-odds scale:

Logodds_scale

$\endgroup$
8
  • 1
    $\begingroup$ Whilst you answer is great, I strongly advise against a model with an interaction term. The interaction coefficient can only reasonably interpreted with either hours or course at zero. This is not a possible option in this very case. $\endgroup$
    – Rachel
    Commented Dec 22, 2019 at 15:36
  • $\begingroup$ Is there a different approach or change to this approach you would suggest @Rachel? $\endgroup$
    – Astarno
    Commented Dec 22, 2019 at 15:50
  • 1
    $\begingroup$ @Astarno given the information at hand, I’d suggest you stick with two models and a analyze the marginal effects. The answer above is pin point accurate. But from a practical point of view, interaction terms are hard to interpreted in your case. $\endgroup$
    – Rachel
    Commented Dec 22, 2019 at 15:53
  • 1
    $\begingroup$ Sorry, I am not answering precisely enough. I only bring across the point that interpreting interaction terms can be troublesome - especially in logit models. In particular, the single coefficient effects are bound to certain limitations - namely b1 shows the effect on the result when b2 is zero, and vice versa. In this case, Hours may be zero. The Training may not - there are no data pints for no training. You may read up on it here: scholar.google.de/… $\endgroup$
    – Rachel
    Commented Dec 22, 2019 at 18:04
  • 1
    $\begingroup$ I don’t see how you can advise against including an interaction without evaluating whether it contributes to the goodness of fit of the model. Yes, an interaction complicates things, but if it’s needed, you shouldn’t just pretend it isn’t there. $\endgroup$
    – Russ Lenth
    Commented Dec 22, 2019 at 22:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.