Multi-level regression interpretation: questio of statistical significance

Question

I am running (multi-level) logit models on hospital data testing whether the ratio two hospital tariffs has any effect on the probability of being admitted to the hospital.

My models are the following:

f1 <- glm(admit ~  log_ratio, data=dat, family = "binomial")
f2 <- glm(admit ~  log_ratio*as.factor(Condition2), data=dat, family = "binomial")
f3 <- lmer(admit ~  log_ratio + (1|provider_id), data=dat_first, REML = F)
f4 <- lmer(admit ~  log_ratio + log_ratio:as.factor(Condition2) + (1|provider_id), data=dat, REML = F)
f5 <- lmer(admit ~  log_ratio*as.factor(Condition2) + (1|provider_id), data=dat, REML = F)

where admit is a binary variable indicating hospital admission, log_ratio is the log-transformed ratio of the two tariffs, Condition2 is the condition for which the patient received treatment and provider_id is the id of the treatment provider.

My results are the following:

============================================================================================================
                                                                   Dependent variable:                      
                                              --------------------------------------------------------------
                                                                          admit                             
                                                     logistic                         linear                
                                                                                  mixed-effects             
                                                  (1)         (2)         (3)          (4)          (5)     
------------------------------------------------------------------------------------------------------------
log_ratio                                      0.605***     1.385*      0.012***     0.013***      0.012    
                                                (0.058)     (0.808)     (0.001)      (0.002)      (0.013)   
                                                                                                            
as.factor(Condition2)Brain Disorder                          1.392                                 -0.001   
                                                            (1.304)                               (0.021)   
                                                                                                            
as.factor(Condition2)Amputation                              2.352                                 0.0002   
                                                            (2.257)                               (0.072)   
                                                                                                            
as.factor(Condition2)Chronic pain                           4.403**                                -0.026   
                                                            (2.161)                               (0.055)   
                                                                                                            
as.factor(Condition2)Nervous system                          2.391                                 0.006    
                                                            (1.551)                               (0.026)   
                                                                                                            
as.factor(Condition2)Organ Disorder                          1.114                                 -0.018   
                                                            (1.787)                               (0.043)   
                                                                                                            
log_ratio:as.factor(Condition2)Brain Disorder               -0.098                   0.014***      0.014    
                                                            (0.825)                  (0.002)      (0.014)   
                                                                                                            
log_ratio:as.factor(Condition2)Amputation                   -0.067                   0.053***      0.053    
                                                            (1.741)                  (0.003)      (0.059)   
                                                                                                            
log_ratio:as.factor(Condition2)Chronic pain                 -1.926                   0.009***      0.024    
                                                            (1.230)                  (0.001)      (0.030)   
                                                                                                            
log_ratio:as.factor(Condition2)Nervous system               -1.537                   0.005***     -0.0003   
                                                            (1.050)                  (0.001)      (0.018)   
                                                                                                            
log_ratio:as.factor(Condition2)Organ Disorder                0.457                   0.023***      0.037    
                                                            (1.242)                  (0.001)      (0.032)   
                                                                                                            
Constant                                       -4.476***   -6.550***    0.022***      0.007        0.008    
                                                (0.075)     (1.293)     (0.008)      (0.009)      (0.023)   
                                                                                                            
------------------------------------------------------------------------------------------------------------
Observations                                    115,376     115,376     115,376      115,376      115,376   
Log Likelihood                                -12,589.260 -12,229.590  56,636.920   56,935.280   56,935.630 
Akaike Inf. Crit.                             25,182.530  24,483.190  -113,265.800 -113,852.600 -113,843.300
Bayesian Inf. Crit.                                                   -113,227.200 -113,765.700 -113,708.100
============================================================================================================
Note:                                                                            *p<0.1; **p<0.05; ***p<0.01

My main interest is in log_ratio and whether it has any association with the probability of hospital admission. My secondary interest is whether this correlation is different per condition. Log-ratio is significant in every model (logit or multi-level logit) except for when I do an interaction effect with condition (with all main effects in the model, such as in Models 2 & 5). The variables with log-ratio then lose their significance.

My question is which model do I believe? My gut feeling is that log-ratio is in fact significant, but due to small sample size per condition or some other reason it is not showing up as significant in Models 2 & 5. Could this be true?

Also, could Model 4 be an acceptable specification of the model? That is, in my case, do I need to have all main effects in the model?

Answer 1

Building off Matt Barstead's comment, looking at the fit information of model 3 vs model 4, I see no evidence that model 4 is the better model. The log likelihood is higher (with 5 additional degrees of freedom used), both AIC and BIC are higher, suggesting that model 4 is not doing a better job of prediction. You can use R's anova() function to test the two models. A quick and dirty calculation suggests that model 4 does not improve fit:

> 1-pchisq((-2*56636.920)-(-2*56935.280), 5)
[1] 0

Because the log likelihood of the more parsimonious model is smaller than the log likelihood from the less parsimonious model in your case, the chi-square test value of 0 is saying the more complicated model is no better than the less complicated model. See information on these tests here.