# Multi-level regression interpretation: questio of statistical significance

I am running (multi-level) logit models on hospital data testing whether the ratio two hospital tariffs has any effect on the probability of being admitted to the hospital.

My models are the following:

f1 <- glm(admit ~  log_ratio, data=dat, family = "binomial")
f2 <- glm(admit ~  log_ratio*as.factor(Condition2), data=dat, family = "binomial")
f3 <- lmer(admit ~  log_ratio + (1|provider_id), data=dat_first, REML = F)
f4 <- lmer(admit ~  log_ratio + log_ratio:as.factor(Condition2) + (1|provider_id), data=dat, REML = F)
f5 <- lmer(admit ~  log_ratio*as.factor(Condition2) + (1|provider_id), data=dat, REML = F)


where admit is a binary variable indicating hospital admission, log_ratio is the log-transformed ratio of the two tariffs, Condition2 is the condition for which the patient received treatment and provider_id is the id of the treatment provider.

My results are the following:


============================================================================================================
Dependent variable:
--------------------------------------------------------------
logistic                         linear
mixed-effects
(1)         (2)         (3)          (4)          (5)
------------------------------------------------------------------------------------------------------------
log_ratio                                      0.605***     1.385*      0.012***     0.013***      0.012
(0.058)     (0.808)     (0.001)      (0.002)      (0.013)

as.factor(Condition2)Brain Disorder                          1.392                                 -0.001
(1.304)                               (0.021)

as.factor(Condition2)Amputation                              2.352                                 0.0002
(2.257)                               (0.072)

as.factor(Condition2)Chronic pain                           4.403**                                -0.026
(2.161)                               (0.055)

as.factor(Condition2)Nervous system                          2.391                                 0.006
(1.551)                               (0.026)

as.factor(Condition2)Organ Disorder                          1.114                                 -0.018
(1.787)                               (0.043)

log_ratio:as.factor(Condition2)Brain Disorder               -0.098                   0.014***      0.014
(0.825)                  (0.002)      (0.014)

log_ratio:as.factor(Condition2)Amputation                   -0.067                   0.053***      0.053
(1.741)                  (0.003)      (0.059)

log_ratio:as.factor(Condition2)Chronic pain                 -1.926                   0.009***      0.024
(1.230)                  (0.001)      (0.030)

log_ratio:as.factor(Condition2)Nervous system               -1.537                   0.005***     -0.0003
(1.050)                  (0.001)      (0.018)

log_ratio:as.factor(Condition2)Organ Disorder                0.457                   0.023***      0.037
(1.242)                  (0.001)      (0.032)

Constant                                       -4.476***   -6.550***    0.022***      0.007        0.008
(0.075)     (1.293)     (0.008)      (0.009)      (0.023)

------------------------------------------------------------------------------------------------------------
Observations                                    115,376     115,376     115,376      115,376      115,376
Log Likelihood                                -12,589.260 -12,229.590  56,636.920   56,935.280   56,935.630
Akaike Inf. Crit.                             25,182.530  24,483.190  -113,265.800 -113,852.600 -113,843.300
Bayesian Inf. Crit.                                                   -113,227.200 -113,765.700 -113,708.100
============================================================================================================
Note:                                                                            *p<0.1; **p<0.05; ***p<0.01


My main interest is in log_ratio and whether it has any association with the probability of hospital admission. My secondary interest is whether this correlation is different per condition. Log-ratio is significant in every model (logit or multi-level logit) except for when I do an interaction effect with condition (with all main effects in the model, such as in Models 2 & 5). The variables with log-ratio then lose their significance.

My question is which model do I believe? My gut feeling is that log-ratio is in fact significant, but due to small sample size per condition or some other reason it is not showing up as significant in Models 2 & 5. Could this be true?

Also, could Model 4 be an acceptable specification of the model? That is, in my case, do I need to have all main effects in the model?

• The question of “which” model you believe is not straightforward. You can compare some of the models using the anova() function. You could keep a hold out sample and test each model on the out-of-sample set. If your interaction term is significant then you probably keep it in the model and probe for which conditions the association is significant. – Matt Barstead Dec 26 '20 at 1:07

Building off Matt Barstead's comment, looking at the fit information of model 3 vs model 4, I see no evidence that model 4 is the better model. The log likelihood is higher (with 5 additional degrees of freedom used), both AIC and BIC are higher, suggesting that model 4 is not doing a better job of prediction. You can use R's anova() function to test the two models. A quick and dirty calculation suggests that model 4 does not improve fit:
> 1-pchisq((-2*56636.920)-(-2*56935.280), 5)