2
$\begingroup$

I am running cluster analysis (using mclust in R) and then looking to see whether various known data groupings (based on metadata) reveals associations between clusters and metadata groups. For each cluster/groupings table I am using the chi squared test, and since there are multiple ways to group the data I am simply using a Bonferroni correction on the resulting p-values.

I've had a hard time finding good references for how to interpret the individual associations, but my understanding currently is that I can determine the significance of a given association by converting the standardized residual of each cell to a p-value using the normal distribution (p = 2*pnorm(-abs(stdres))). Is this correct, and further is it proper to not adjust these values based on the size of the given table? I've seen nothing to suggest I should perform such a correction, so I'm assuming it is unnecessary.

If so, then given the fact that I am testing multiple tables, should I be simply multiplying each cell's p-value by the number of tables to achieve the right multiple testing correction?

Improve this question
$\endgroup$
2
  • 1
    $\begingroup$ If you're formally trying to test them, maybe. If you're just saying 'where were the big ones?' then probably not. $\endgroup$
    – Glen_b
    Jun 7, 2013 at 3:55
  • $\begingroup$ For the larger table we're just using the test to determine which data to drill down into, in which case I agree that it is not so important to correct perfectly. However if there were many cells with modest N then the proper correction could make a big difference in detecting multiple real effects. My thought right now is to multiply each cell's p-value by the table's degrees of freedom. That still might be conservative, but I don't believe it would be too liberal a correction. $\endgroup$
    – Todd Gillette
    Jun 13, 2013 at 19:17

1 Answer 1

Reset to default
2
$\begingroup$

This looks correct to me but I am not so sure about your view that no adjustment is necessary. Obviously, if you examine many cells, you will have an inflated type I error rate. I don't see any particular reason why this would not matter at all and we could happily ignore multiple testing issues in this particular setting but not in others.

At the same time, a Bonferroni correction would be very conservative and since we only look at residuals after rejecting the null hypothesis of independence over the whole table, we already know that there must be some pattern of association somewhere.

One way out of this dilemma is to give up rigid binary interpretations but if you insist on formal testing then I would think that you also need to adjust those $p$-values for the size of the table.

Agresti (Categorical Data Analysis) discusses standardized residuals as a follow-up to the chi-squared test of independence in two-way contingency tables on pp. 80-81 and has this to say:

A standardized Pearson residual that exceeds about 2 or 3 in absolute value indicates lack of fit of $H_0$ in that cell. Larger values are more relevant when df is larger and it becomes more likely that at least one is large simply by chance.

I take this to mean that, strictly speaking, adjusting these $p$-values would in fact be necessary (as intuition would suggest) but that he prefers an informal approach to the problem (in a standard normal deviate, 2 is roughly $p$ = .05, two-tailed, but he clearly implies that this particular threshold should not be interpreted too literally).

Improve this answer
$\endgroup$
1
  • $\begingroup$ Thank you greatly. Finally a reference that raises the issue of table size (df) and cell significance. Since I had seen no mention previously I thought perhaps something was happening in the test calculations that accounted for the multiple cells. I just wish I had some sense of approximately how large a correction would be appropriate, even without an exact value. Just df, or maybe closer to df/2? $\endgroup$
    – Todd Gillette
    Jun 6, 2013 at 22:16

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.