Understanding regression results when data are subsetted

Question

I have some data that span several years: 2006-2010. I have run logistic regression to model the data. For the whole dataset, I get a 95% confidence interval for the odds ratio of a parameter of interest of

(0.34 - 0.47 ) 

indicating a very significant effect. However, for each individual year, with the same model specification, the confidence intervals are:

2006: (0.78 - 1.94) (not significant)
2007: (0.61 - 0.93)
2008: (0.63 - 0.90)
2009: (0.92 - 1.30) (not significant)
2010: (0.88 - 1.33) (not significant)

How can I reconcile that the confidence interval for the whole dataset is below the confidence intervals for all the individual years ? I'm guessing that it is to do with the sample sizes - bigger sample sizes leading to lower p values: I think I get the maths behind this, but I can't get the intuition behind it.

Update1

In response to the answers by Michael and Peter, I am providing more information.

The model is:

death~treatment+age+imd+smoking+clinicals+drugs+comorbidities

Notes:

1. death is binary
2. treatment is binary - whether treatment A or treatment B was given. This is the parameter of interest that I have given confidence intervals for (which I obtained by exponentiating the CIs for the estimate +/- 1.96xSE )
3. age is age in years
4. imd is a socioeconomic status index for the patient
5. smoking is categorical and has several levels pertaining to the patients smoking status.
6. clinicals is a set of clinical measures such as heartrate, blood pressure
7. drugs is the a set of binary covariates indicating whether a particular drug was given
8. comorbidites is a set of binary covariates indicating whether the patient is suffering from certain conditions: eg asthma, diabetes

In the overall model I have not included year as a covariate - the same model formula was used for the subsetted data and the whole data.

There is no problem with collinearity between the continuous variables but I am less sure about associations between the categorical variables. I think this could be a problem but I don't know how to tackle it - I tried some chi-square tests but nothing was independent from anything else (I thought that might be due to the sample size - according to my teacher it doesn't make sense that asthma would be collinear with diabetes for instance)

Update2

After further comments by Michael I am now giving some more info....The ratio of treatment A/B has changed a lot over the period - it was a new treatment in 2006 and hardly used, but is now the treatment of choice:

2006: 555 out of 11,505
2007: 2,810 out of 12,307
2008: 5,669 out of 13,243
2009: 9,111 out of 14,654
2010: 12,368 out of 15,573

Overall: 30,643 out of 92,767

The death rate has not changed much (around 7% throughout)

Answer 1

To expand on Peter Flom's answer (which is echoed in Michael Chernick's subsequent reply), this graphic may help the intuition.

The following R code shows how it was produced. Briefly, it generates 400 data points per year, with values of variable $x$ ranging variously from $0$ to $2$ through $2$ to $4$, shifting upwards each year: this is one (mild) form of confounding of $x$ and year. It creates probabilities $y$ according to a logistic model and, for later use, generates binary observations $z$ according to those probabilities. The probabilities are plotted, with color differentiating the years. Finally, the logistic fit to those 2000 observations is overplotted with a dashed line.

# Create sample data
#
logistic <- function(x) 1 / (1 + exp(-x))
n <- 400      # Values per year
offset <- 80  # Shift in x-values year-to-year
year <- as.factor(floor(seq(from=2006,to=2011-1/n, by=1/n)))
x <- as.vector(sapply(1:5, function(i) offset*i + 1:n)) / n * 2
y <- logistic((4 - 2/3 * x - unclass(year)))
set.seed(17)
z <- rbinom(length(y), 1, prob=y)
data <- data.frame(cbind(x, year, y, z))
#
# Plot the *probabilities* which underlie the data.
#
par(mfrow=c(1,1))
plot(x, y, col=year, pch=19, cex=0.75, ylab="Probability", 
     main="Individual and Overall Fits")
#
# Plot the overall fit.
#
b <- summary(glm(z ~ x, data=data, family=binomial(link="logit")))$coefficients
curve(logistic(b[1] + b[2]*x), add=TRUE, lwd=3, lty=2, col="Gray")

Apparently, the effect of this "staircase" of falling curves is to cause the overall model to average its way through the middle, suggesting a much steeper fall than any individual step (year) exhibits.

We can do the calculations of the odds ratios and their 95% CIs, too:

output <- function(d) {
  b <- summary(glm(z ~ x, data=d, family=binomial(link="logit")))$coefficients
  a <- b["x", "Estimate"]
  u <- b["x", "Std. Error"]
  z <- qnorm(c(.025, .975))
  exp(c(a, a + z*u))
}
output(data)
by(data, data$year, output)

The (cleaned up) output, which is in the form (estimate, lower limit, upper limit) for each model, is

Overall
0.17 0.14 0.20
year: 1
0.54 0.31 0.94
year: 2
0.53 0.37 0.78
year: 3
0.36 0.25 0.53
year: 4
0.61 0.36 1.03
year: 5
0.47 0.22 1.01

The overall coefficient of 0.17 is lower than the lower confidence limits of any of the annual (year-specific) fits, several of which are evidently not even significant (because their confidence intervals include 1.0).

---

Comments

This phenomenon is not special to logistic regression: it is seen in many ordinary regressions and, in that form, has been discussed and illustrated elsewhere on this site. (Finding that discussion might take some clever searching, I'm afraid.) Many textbooks use examples like this to illustrate the value of including and controlling for important variables in statistical models and to discuss confounding.

This example uses fairly large (sub)sample sizes: this suggests that the changes in widths of confidence intervals are not wholly due to changes in sample size. In fact, most of the change is because each "stair" is much shallower than the overall trend, and therefore is not as easy to discriminate from no trend at all.

There is no multiple testing going on: we're looking only at one covariate.

Answer 2

In this case, it does not appear to have to do with sample sizes, since the CIs for the individual years do not even overlap with the CI for the whole period.

It's hard to say exactly what's going on. Your code would help - did the model for the full data set include year as a IV? What is your dependent variable? What is your independent variable?

It certainly seems that year is a confounding variable. In other words, you have

DV ~ IV

but, in addition, DV is related to year and IV is related to year.

Confounds with time are pretty common. If DV becomes more likely over time, and IV increases with time, then a set of relationships like the one you found would exist.

Answer 3

I think the smaller sample size explains why some years are siginficant and others are not. Actually if you do multiplicity correction for doing 5 different test you may find that none of them are significant given a proper p-value adjustment to the tests.

But Peter has hit on an important observation. The individual years give odds ratios that are close to 1 (three cases 1 is included in the interval). But when the years are pooled the ratio is much further away from 1 compared to any of the individual years. This suggests to me that some factor influencing the outcome differs from year to year. If that is the case the data is really not poolable without inclusion of this confounding factor as another covariate in the model. However to make sense of this we would need to know more specifics. You haven't told us what covariate you are using and the values of the covariate that you are comparing to get an odds ratio. If we know that and more about the nature of the data and your problem it may be possible to figure out what the confounding covariate is and whether or not it is observable and can therefore be used as a covariate in the model.

But the answer to your question about intuition, the results you see are due to differences in sample size, multiple testing and some cause for the model to depend on the year and hence not be poolable.