# Confidence & conclusion in rare event analysis

I am looking at the http://www.nejm.org/doi/full/10.1056/NEJMoa021134 paper and don't understand how they made their conclusions.

According to the data in Table 1 they analyzed 440655 vaccinated children and 96648 unvaccinated children.

As of vaccinated childred 269 got Autism, 352 got other ASD, total 621, giving 6.1, 8.0 and 14.1 total cases per 10000 vaccinated children.

As of unvaccinated childred 47 got Autism, 70 got other ASD, total 117, giving 4.9, 7.2 and 12.1 total cases per 10000 unvaccinated children.

So, in conrast to their conslusion I can see vaccinated chilren got higher autism & ASD rate and the question is level of confidence.

Using http://www.quantpsy.org/chisq/chisq.htm calculator I calculate three p-values (Autism vs Other, ASD vs Other, Total vs Other) using two Conditions: Vaccinated and Unvaccinated and get this numbers:

Test            | Chi-square | p-value    |
Autism vs Other | 2.079      | 0.14933798 |
ASD vs Other    | 0.561      | 0.45385761 |
Total vs Other  | 2.281      | 0.13096741 |


As far as I understand, this means that there is some, but low confidence ( p-value < 0.15 ) for the Autism and Total cases.

Also there is a second table, talking about person-years, total 2129864 years total, 482360 unvaccinated, 1647504 vaccinated.

For vaccinated it's 263 Autism cases and 345 other ASD cases, 608 total, giving 1.6 and 2.1, 3.7 total cases per 10000 person-years.

For unvaccinated it's 53 Autism cases and 77 other ASD cases, 130 total, giving 1.1 and 1.6, 2.7 total cases per 10000 person-years.

The difference is event more visible and here is Chi-square and p-value data (same calculator):

Test            | Chi-square | p-value    |
Autism vs Other | 6.228      | 0.01257457 |
ASD vs Other    | 4.667      | 0.03074759 |
Total vs Other  | 12.215     | 0.00047407 |


So, here the confidence is strong enough for all 3 cases ( p < 0.05). For total case it's very strong ( p < 0.0005 ). Yet I am not sure if Chi-square analysis is applicable for person-years case.

And at the same time paper concludes that there is no dependency between the vaccination and Autism / ASD.

Am I missing something in the analysis?

The study you cite necessarily went a good deal farther than simply counting up the number of cases, as you did. It took into account the age at which a child received MMR vaccine (counting the child as unvaccinated up until that point), the age at vaccination, the time since vaccination, the year when the vaccine was given, and the age at diagnosis, in a time-dependent model of incidence-rate ratios. It also corrected for the child's gestational age at birth, birth weight, sex, mother's education, and socioeconomic status. Those types of corrections are very important in this type of population health study, as those other factors might also be associated with the outcome of interest (here, risk of autism or other ASD).

When those factors were correctly taken into account, the relative risk of "autistic disorder" was nominally lower in those that were vaccinated, by a factor of 0.92; this was not, however, significantly different statistically from a relative risk of 1 (or no difference in risk). Similarly, the relative risk of other ASD was a (non-significant) factor of 0.83 in the vaccinated versus unvaccinated.

So the main things you are "missing" is that the study's statistical analysis examined relative risks of developing autism or ASD in time-dependent models, not just the yes/no of whether these occurred, and that other critical variables were properly considered.

• So, you are saysing that while the data published shows increased rate of autism by ~20% in both tables 1 and 2, some analysis not supported by the data published says otherwise? How can this be verified? Jun 3 '16 at 21:52
• Basically, this means this 20% were adjusted while adjusting for some other factor like socioeconomic status or gestational age. It's very interesting to know which factor accounted for this taken amount of data collected and the fact that even nowadays noone knows real causes of Autism. Jun 3 '16 at 22:05
• BTW: The paper says that "First, the risk of autism was similar in vaccinated and unvaccinated children, in both age-adjusted and fully adjusted analyses." It's hard to say what age is meant here. I suppose it's child age at time of research, because vaccination age is not applicable to non-vaciinated children and diagnosis age is not applicable to healthy children. Basically, it should mean they have very different age distribution for healthy and dignosed children to account for 20% difference. It's rather strange given the study completeness of the data Jun 3 '16 at 22:27
• These are longitudinal studies based on a nation with relatively complete health records from birth. Note in Table 1 that birth weight, gestational age at birth, socioeconomic status, and mother's education were all associated with vaccination status. Examine the manual for the SAS PROC GENMOD log-linear Poisson function they used, to get an idea about correction for covariates. For further details you would have to consult with the authors of the study.
– EdM
Jun 3 '16 at 22:41

P-value isn't a scale for correlation. For example, you mention "As far as I understand, this means that there is some, but low confidence ( p-value < 0.15 ) for the Autism and Total cases."

That's not how to interpret p-value. You need to understand the following:

High P values: your data are likely with a true null. Low P values: your data are unlikely with a true null.

So the closer you get to 0, your data will prove that your hypothesis is unlikely to be true. so 0.15 means that there is very little chance that the two are correlated.

• Sorry, but could you reference to something to support your words? I far as I can see even from wikipedia en.wikipedia.org/wiki/Null_hypothesis "Rejecting or disproving the null hypothesis—and thus concluding that there are grounds for believing that there is a relationship between two phenomena (e.g. that a potential treatment has a measurable effect)—is a central task in the modern practice of science, and gives a precise criterion for rejecting a hypothesis". I rejected the Null hypothesis with threshold 0.15 and so it means that there is correlation (with given confidence). Jun 3 '16 at 18:22
• Also it corresponds with simple test: When I put 10x numbers into the calculator my Chi-square raises and p-value lowers, giving more confidence that this 20% difference is not just a fluctuation. More tests -> more confidence, this is expected. Also when I put exactly same values into two conditions I receive Chi-square 0 and p-value 1, that means "no difference between conditions at all", that is also expected. Jun 3 '16 at 18:28