Two models estimated using two different, but strongly related predictors (r > 0.99) yielding different results

This isn't a problem with correlation between predictors - I have two models, each considers only one of the variables. That is the only difference between the models.

I'm estimating the probability of an diagnosis given some confounders and a measure of monthly temperature. I have two possible temperature definitions I'm considering: monthly average temperature and monthly average high temperature. I don't expect the response to temperature to be linear, so I broke average temperature into 5 degree bins with bottom and top coding at < 40 and > 90. I did the same with average high temperature but shifted the bins slightly with bottom and top coding < 50 and > 100.

I estimate the first logistic model

event ~ age + sex + ... + mean_temp_group


and get the response I'd expect from my theorized process. However, I'd prefer to report the results using mean high temperature since average temperature is misleadingly low (average temp of 70, for instance, is pretty warm with highs in the 80s but people think "70 degrees? That's wonderful!"). So I estimate the same model but instead replace mean_temp_group with mean_high_group:

event ~ age + sex + ... + mean_high_group


and the results don't match either my theory or what I saw with mean_temp_group.

That seems weird given how similar the two variables are. The average and average high variables have a correlation coefficient of 0.9939. In essence the average high is the average plus a constant (on average, 9.4 degrees with a standard deviation of 2.1).

At first I assumed this was a problem with the code, so I re-pulled the data (still have the same problem and the data extraction seems to be accurate). I also took the model with mean_temp_group and edited the formula in place to read mean_high_group lest I omitted/included a different variable between the models (I didn't).

I assume it has something to do with the binning or something along those lines - any ideas? I'm very confused by two variables that basically appear to be an additive shift of each other giving very different results.

• Please show us exactly how the results don't match. Are you referring to predicted log odds? The fitted coefficients? Something else? What regression diagnostics have you run?
– whuber
Commented Aug 12, 2016 at 20:56
• "I don't expect the response to temperature to be linear" — Then use polynomial terms or a spline. Discretizing the temperature doesn't make a lot of sense. Commented Aug 12, 2016 at 21:21
• @whuber, the odd ratios in the model with average temperature are ~1 from 40-60 and then trend upwards maxing out at 1.15-1.20. They are ~1 for the entire range for the model with the average high temperature. I'm open to any diagnostics you have in mind. Commented Aug 12, 2016 at 21:27
• @Kodiologist, I've been binning temperature and fitting fixed effects to get the flexibility that comes without imposing a functional form on the response. Normally, the chief trade-off with this approach is estimating a lot of parameters with a relatively small data set. In this case, I'm trying to estimate about 40 parameters with 55,000,000 observations and don't suffer as directly from that problem. I've considered estimating with something like GAM smoother but such an approach doesn't lend itself to hypothesis testing. Commented Aug 12, 2016 at 21:35
• But you are imposing a functional form on the response, namely, a piecewise constant function with predefined breaks. The problem is not that discretization creates a lot of parameters but that it's extremely coarse. See also Frank Harrell's piece here. Commented Aug 12, 2016 at 21:38

1 Answer

Just some speculations. If we plot the coefficients of binned high temperature versus those of binned mean temperature we can see a curve linear relationship:

My guesses are:

i) While the correlation is high, the very high number of sample size might have masked some subtle non-linear relationship. In some climates, it's possible that there is a wider variability in summer compared to winter. In that sense, it's not like all cases are just moved to the next bin, there could have been some reshuffling, causing the difference.

ii) Following up with the point above, in the graph we can also see the associations with the outcome seems to be diminishing compared to that of the mean temperature. A possibility is that at a very high temperature (e.g. when a heat wave strikes,) people might change behaviors that mitigated their risk. For instance they may have stayed indoor and turned on the AC, so that they were less likely to get heat stroke or dehydration, etc.

And I think that the 5-degree binning approach seems a bit wasteful. With so many data points it may be worth to examine the association between temperature and the log odds of the events (can be done in single degree), and evaluate if binning really needs to be done, or can temperature be included in some other functional forms.

• There is definitely behavior change driving the noise at the high temperatures (combined with the relatively small n at the edges). I went through and estimated the model on a 5% subset with 1-degree bins and that results in the two models giving similar results. I'm going to attempt to fit the model using the full dataset within 1-degree bins. The reshuffling idea seems to potentially explain the problem with the larger 5-degree bins. Commented Aug 15, 2016 at 16:14