# Understanding regression results when data are subsetted

I have some data that span several years: 2006-2010. I have run logistic regression to model the data. For the whole dataset, I get a 95% confidence interval for the odds ratio of a parameter of interest of

(0.34 - 0.47 )


indicating a very significant effect. However, for each individual year, with the same model specification, the confidence intervals are:

2006: (0.78 - 1.94) (not significant)
2007: (0.61 - 0.93)
2008: (0.63 - 0.90)
2009: (0.92 - 1.30) (not significant)
2010: (0.88 - 1.33) (not significant)


How can I reconcile that the confidence interval for the whole dataset is below the confidence intervals for all the individual years ? I'm guessing that it is to do with the sample sizes - bigger sample sizes leading to lower p values: I think I get the maths behind this, but I can't get the intuition behind it.

Update1

The model is:

death~treatment+age+imd+smoking+clinicals+drugs+comorbidities


Notes:

1. death is binary
2. treatment is binary - whether treatment A or treatment B was given. This is the parameter of interest that I have given confidence intervals for (which I obtained by exponentiating the CIs for the estimate +/- 1.96xSE )
3. age is age in years
4. imd is a socioeconomic status index for the patient
5. smoking is categorical and has several levels pertaining to the patients smoking status.
6. clinicals is a set of clinical measures such as heartrate, blood pressure
7. drugs is the a set of binary covariates indicating whether a particular drug was given
8. comorbidites is a set of binary covariates indicating whether the patient is suffering from certain conditions: eg asthma, diabetes

In the overall model I have not included year as a covariate - the same model formula was used for the subsetted data and the whole data.

There is no problem with collinearity between the continuous variables but I am less sure about associations between the categorical variables. I think this could be a problem but I don't know how to tackle it - I tried some chi-square tests but nothing was independent from anything else (I thought that might be due to the sample size - according to my teacher it doesn't make sense that asthma would be collinear with diabetes for instance)

Update2

After further comments by Michael I am now giving some more info....The ratio of treatment A/B has changed a lot over the period - it was a new treatment in 2006 and hardly used, but is now the treatment of choice:

2006: 555 out of 11,505
2007: 2,810 out of 12,307
2008: 5,669 out of 13,243
2009: 9,111 out of 14,654
2010: 12,368 out of 15,573


Overall: 30,643 out of 92,767

The death rate has not changed much (around 7% throughout)

-

To expand on Peter Flom's answer (which is echoed in Michael Chernick's subsequent reply), this graphic may help the intuition.

The following R code shows how it was produced. Briefly, it generates 400 data points per year, with values of variable $x$ ranging variously from $0$ to $2$ through $2$ to $4$, shifting upwards each year: this is one (mild) form of confounding of $x$ and year. It creates probabilities $y$ according to a logistic model and, for later use, generates binary observations $z$ according to those probabilities. The probabilities are plotted, with color differentiating the years. Finally, the logistic fit to those 2000 observations is overplotted with a dashed line.

# Create sample data
#
logistic <- function(x) 1 / (1 + exp(-x))
n <- 400      # Values per year
offset <- 80  # Shift in x-values year-to-year
year <- as.factor(floor(seq(from=2006,to=2011-1/n, by=1/n)))
x <- as.vector(sapply(1:5, function(i) offset*i + 1:n)) / n * 2
y <- logistic((4 - 2/3 * x - unclass(year)))
set.seed(17)
z <- rbinom(length(y), 1, prob=y)
data <- data.frame(cbind(x, year, y, z))
#
# Plot the *probabilities* which underlie the data.
#
par(mfrow=c(1,1))
plot(x, y, col=year, pch=19, cex=0.75, ylab="Probability",
main="Individual and Overall Fits")
#
# Plot the overall fit.
#
b <- summary(glm(z ~ x, data=data, family=binomial(link="logit")))$coefficients curve(logistic(b[1] + b[2]*x), add=TRUE, lwd=3, lty=2, col="Gray")  Apparently, the effect of this "staircase" of falling curves is to cause the overall model to average its way through the middle, suggesting a much steeper fall than any individual step (year) exhibits. We can do the calculations of the odds ratios and their 95% CIs, too: output <- function(d) { b <- summary(glm(z ~ x, data=d, family=binomial(link="logit")))$coefficients
a <- b["x", "Estimate"]
u <- b["x", "Std. Error"]
z <- qnorm(c(.025, .975))
exp(c(a, a + z*u))
}
output(data)
by(data, data$year, output)  The (cleaned up) output, which is in the form (estimate, lower limit, upper limit) for each model, is Overall 0.17 0.14 0.20 year: 1 0.54 0.31 0.94 year: 2 0.53 0.37 0.78 year: 3 0.36 0.25 0.53 year: 4 0.61 0.36 1.03 year: 5 0.47 0.22 1.01  The overall coefficient of 0.17 is lower than the lower confidence limits of any of the annual (year-specific) fits, several of which are evidently not even significant (because their confidence intervals include 1.0). ### Comments This phenomenon is not special to logistic regression: it is seen in many ordinary regressions and, in that form, has been discussed and illustrated elsewhere on this site. (Finding that discussion might take some clever searching, I'm afraid.) Many textbooks use examples like this to illustrate the value of including and controlling for important variables in statistical models and to discuss confounding. This example uses fairly large (sub)sample sizes: this suggests that the changes in widths of confidence intervals are not wholly due to changes in sample size. In fact, most of the change is because each "stair" is much shallower than the overall trend, and therefore is not as easy to discriminate from no trend at all. There is no multiple testing going on: we're looking only at one covariate. - Nice example, I've made similar charts before to demonstrate aggregation bias, or in particular for multi-level data how individual and contextual effects are confounded when you aggregate data. – Andy W Sep 21 '12 at 0:01 +! @whuber you go to a lot of pains to make really nice simulationa and graphic illustrations. If the OP had trouble understanding parts of my answer and Peter's This certainly shoyuld make things clear. I think my discussion with the OP which you may have seen all or most of, indicated that when he adjusted the pooled data the the ratios were very similar. The other issue is that he compared at one specific point on the regression curve and it was not clear that all the curves used truly comparble points. – Michael Chernick Sep 21 '12 at 1:37 +1 !! Thank you - this is a very helpful and useful...... – Joe King Sep 21 '12 at 21:42 add comment In this case, it does not appear to have to do with sample sizes, since the CIs for the individual years do not even overlap with the CI for the whole period. It's hard to say exactly what's going on. Your code would help - did the model for the full data set include year as a IV? What is your dependent variable? What is your independent variable? It certainly seems that year is a confounding variable. In other words, you have DV ~ IV but, in addition, DV is related to year and IV is related to year. Confounds with time are pretty common. If DV becomes more likely over time, and IV increases with time, then a set of relationships like the one you found would exist. - Thank you +1 !!! I have updated the question with more details – Joe King Sep 20 '12 at 17:23 FYI, I have updated the question again with even more details. Thanks ! – Joe King Sep 20 '12 at 17:47 add comment I think the smaller sample size explains why some years are siginficant and others are not. Actually if you do multiplicity correction for doing 5 different test you may find that none of them are significant given a proper p-value adjustment to the tests. But Peter has hit on an important observation. The individual years give odds ratios that are close to 1 (three cases 1 is included in the interval). But when the years are pooled the ratio is much further away from 1 compared to any of the individual years. This suggests to me that some factor influencing the outcome differs from year to year. If that is the case the data is really not poolable without inclusion of this confounding factor as another covariate in the model. However to make sense of this we would need to know more specifics. You haven't told us what covariate you are using and the values of the covariate that you are comparing to get an odds ratio. If we know that and more about the nature of the data and your problem it may be possible to figure out what the confounding covariate is and whether or not it is observable and can therefore be used as a covariate in the model. But the answer to your question about intuition, the results you see are due to differences in sample size, multiple testing and some cause for the model to depend on the year and hence not be poolable. - – Joe King Sep 20 '12 at 19:51 Okay @JoeKing you set it up and I will go to comment sometime when I get a chance. Iwas hoping this would be quick and have a simple answer. – Michael Chernick Sep 20 '12 at 20:40 I'm not really sure how to use chat. I left a message there for you, but nothing happened so I left another one saying I would try again later. What to do ? Anyway, maybe we can continue in the comments here ? Well, I get what you said about the covariates - but isn't that the whole point of adjusting for them ?...to account for possible confounding ? I used all the same covariates in all the models - except that now I also have a year term in the model that has all the years. Maybe it's simply not possible to compare them ? But at least there is some consistency in the ORs now ! – Joe King Sep 20 '12 at 20:47 I just hadn't checked in there yet. You probably did nothing wrong. I have had discussions in chat and it seems fairly easy to do but I haven't initiated any myself yet. I don't think you understand my question yet. I do not object to comparing the odds ratios but there has to be some common ground. you can do it if you have all the same covariates in the models and fix them to the same values. Then you can see if and how the p$_s\$ change based on the changes in the data sets from year to year and combined. –  Michael Chernick Sep 20 '12 at 21:43
We moved to chat. I saw his later comments. He will not see mine until tomorrow as he said he is in the UK. If you have to delete the comments before tomorrow then I will just repeat and resume in chat. I don't think it is necessary to move the latter comments. @whuber Thanks for letting me know. –  Michael Chernick Sep 20 '12 at 23:51