Confidence & conclusion in rare event analysis

Question

I am looking at the http://www.nejm.org/doi/full/10.1056/NEJMoa021134 paper and don't understand how they made their conclusions.

According to the data in Table 1 they analyzed 440655 vaccinated children and 96648 unvaccinated children.

As of vaccinated childred 269 got Autism, 352 got other ASD, total 621, giving 6.1, 8.0 and 14.1 total cases per 10000 vaccinated children.

As of unvaccinated childred 47 got Autism, 70 got other ASD, total 117, giving 4.9, 7.2 and 12.1 total cases per 10000 unvaccinated children.

So, in conrast to their conslusion I can see vaccinated chilren got higher autism & ASD rate and the question is level of confidence.

Using http://www.quantpsy.org/chisq/chisq.htm calculator I calculate three p-values (Autism vs Other, ASD vs Other, Total vs Other) using two Conditions: Vaccinated and Unvaccinated and get this numbers:

Test            | Chi-square | p-value    |
Autism vs Other | 2.079      | 0.14933798 |
ASD vs Other    | 0.561      | 0.45385761 |
Total vs Other  | 2.281      | 0.13096741 |

As far as I understand, this means that there is some, but low confidence ( p-value < 0.15 ) for the Autism and Total cases.

Also there is a second table, talking about person-years, total 2129864 years total, 482360 unvaccinated, 1647504 vaccinated.

For vaccinated it's 263 Autism cases and 345 other ASD cases, 608 total, giving 1.6 and 2.1, 3.7 total cases per 10000 person-years.

For unvaccinated it's 53 Autism cases and 77 other ASD cases, 130 total, giving 1.1 and 1.6, 2.7 total cases per 10000 person-years.

The difference is event more visible and here is Chi-square and p-value data (same calculator):

Test            | Chi-square | p-value    |
Autism vs Other | 6.228      | 0.01257457 |
ASD vs Other    | 4.667      | 0.03074759 |
Total vs Other  | 12.215     | 0.00047407 |

So, here the confidence is strong enough for all 3 cases ( p < 0.05). For total case it's very strong ( p < 0.0005 ). Yet I am not sure if Chi-square analysis is applicable for person-years case.

And at the same time paper concludes that there is no dependency between the vaccination and Autism / ASD.

Am I missing something in the analysis?

Answer 1

The study you cite necessarily went a good deal farther than simply counting up the number of cases, as you did. It took into account the age at which a child received MMR vaccine (counting the child as unvaccinated up until that point), the age at vaccination, the time since vaccination, the year when the vaccine was given, and the age at diagnosis, in a time-dependent model of incidence-rate ratios. It also corrected for the child's gestational age at birth, birth weight, sex, mother's education, and socioeconomic status. Those types of corrections are very important in this type of population health study, as those other factors might also be associated with the outcome of interest (here, risk of autism or other ASD).

When those factors were correctly taken into account, the relative risk of "autistic disorder" was nominally lower in those that were vaccinated, by a factor of 0.92; this was not, however, significantly different statistically from a relative risk of 1 (or no difference in risk). Similarly, the relative risk of other ASD was a (non-significant) factor of 0.83 in the vaccinated versus unvaccinated.

So the main things you are "missing" is that the study's statistical analysis examined relative risks of developing autism or ASD in time-dependent models, not just the yes/no of whether these occurred, and that other critical variables were properly considered.

Answer 2

P-value isn't a scale for correlation. For example, you mention "As far as I understand, this means that there is some, but low confidence ( p-value < 0.15 ) for the Autism and Total cases."

That's not how to interpret p-value. You need to understand the following:

High P values: your data are likely with a true null. Low P values: your data are unlikely with a true null.

So the closer you get to 0, your data will prove that your hypothesis is unlikely to be true. so 0.15 means that there is very little chance that the two are correlated.