3
$\begingroup$

I have $i>100$ Cox regression models which differ only in one explanatory variable, $x_i$. They have the same outcome data and the same other explanatory (adjustment) variables.

I want to control the global Type 1 error probability/adjust for multiple testing of the significance of the coefficients of the $x_i$.

Some of the $x_i$ show high pairwise correlations. I inferred that therefore their coefficients and p-values are dependent. Neither Bonferroni nor Holm correction take into account these dependencies. The most attractive/exact option in my eyes is a permutation test that does consider them.

I thought about scrambling the rows of the outcome survival matrix, thereby conserving the correlations between the $x_i$ while destroying their relationship with the outcome. Then I compute the coefficients of all the $x_i$. In doing this for a few thousand different resamples I get the multivariate null distribution of the coefficients.

And this is where I'm stuck. I wanted to take the proportion of the null/resampled distribution for each coefficient estimated in the original data that is higher than it (or lower in the case of negative coefficients) as its p-value. But that would, again, destroy the dependency structure, as it treats the multivariate distribution as $i$ separate univariate ones.

How would one go about doing a permutation test for this problem? Is it even possible?

|cite|improve this question
$\endgroup$
2
$\begingroup$

I like the permutation approach, one possibility is to compute the mean vector from your permutation distribution (should be close to a vector of 0's) and the variance/covariance matrix. Then use that information to compute the Mahalanobis distance for all the points, the p-value would then be the proportion of points with a Mahalanobis distance greater than the original coefficients.

|cite|improve this answer
$\endgroup$
  • $\begingroup$ Thank you. If I understand correctly, this would give me one p value expressing the probability of my coefficients given that there is no true relationship of any of the $x_i$ with the outcome? While I find this very interesting, it does not give me the answer I'm looking for; I need to be able to do inference for each coefficient while adjusting for multiple testing. Also I think as some of the coefficients have very low p values even after bonferroni correction, I can expect the global p obtained by the Mahalanobis distance procedure you described to turn out significant too. No? $\endgroup$ – miura Nov 9 '12 at 10:48
  • 2
    $\begingroup$ Yes this gives a single overall p-value. If you only look at the individual p-values after finding this overall one significant then that at least gives a protected test. Are you interested in finding if x1 is significant beyond the contributions of x2? Or finding which of x1 or x2 (which are correlated and therefore the one that does not cause the outcome will still show significance due to the relation with the other) causes the outcome? Or some other question? $\endgroup$ – Greg Snow Nov 9 '12 at 18:54
  • $\begingroup$ I'm interested in the significance level of the coefficients of the $x_i$ after correction for multiple testing. I feel that the higher the correlation between an $x_i$ and another $x_j$, the more a test of the significance of the coefficient of $x_j$ is just a repetition of one of $x_i$, and they should not both contribute 1 to the number of tests a bonferroni or holm procedure corrects for - that would be overly conservative. After giving it some thought, maybe a multivariate procedure naturally considering the correlation structure is superior to my initial approach here. $\endgroup$ – miura Nov 14 '12 at 7:41
  • $\begingroup$ This is very interesting. Let's say the general permutation test is significant, and there are 100 coefficients, would using $\alpha = 0.05$ be ok if we want to find which one of coefficients causing the outcome? Is this what you meant by a protected test? $\endgroup$ – NULL Oct 22 '17 at 1:06
  • 1
    $\begingroup$ @NULL, That is what I mean by a protected test, but that just means that the overall family-wise probability is controlled. It helps a little with the individual tests, but you could still find more significant by chance than just the 5% expected. How many more depends on how many are truly non-zero, which you will not know. $\endgroup$ – Greg Snow Oct 23 '17 at 15:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.