Timeline for Permutation test in multiple linear regression, which way is correct?
Current License: CC BY-SA 4.0
10 events
| when toggle format | what | by | license | comment | |
|---|---|---|---|---|---|
| Nov 10 at 13:49 | comment | added | Mark Nh | @jarbet: My comment above is not accurate. Indeed, there is a clever way to twist bootstrap to prodece distributions of test statistics under the null hypothesis by Hall and Wilson (1991, Two Guidelines for Bootstrap Hypothesis Testing, Biometrics). This is probably what you suggested and it may be a good alternative to the permutation approach! | |
| Nov 4 at 12:36 | comment | added | Mark Nh | @jarbet: Thanks. Bootsrap mimics the sampling distribution of a statistic. But my goal is to simulate the joint distribution of several test statistics under the null hypothesis. This is to achive adjusted p-values for correlated tests. I think I need to use permuation distribution rather than bootstrap to achieve this. | |
| Nov 1 at 20:22 | comment | added | jarbet |
@MarkNh Got it. FWIW, I want to share an alternative bootstrap approach to permutation testing that let's you get confidence intervals and pvalues for each predictor from only one set of bootstrap replications: cran.r-project.org/web/packages/boot.pval/boot.pval.pdf. Check out the boot_summary function which works for glm and other models.
|
|
| Nov 1 at 15:11 | comment | added | Mark Nh | @jarbet: Freedman-Lane approach pertains only to a sigle coefficient. You need to run it separately for each of the expalnatory variables if needed. | |
| Oct 31 at 17:07 | comment | added | jarbet | @MarkNh: Does the Freedman-Lane approach let you get pvalues for each predictor in the model with only one set of permutations, or do you have to do a separate set of permutations for each predictor? Just curious. | |
| Oct 31 at 8:33 | comment | added | Mark Nh | @jarbet: I agree with Kozolovska. But when the interest is in one particular regression coefficient (say that of covariate X1), I think permuting the outcome is not a good option. In that case, we want to perturbe the relationship between the outcome Y and X1, but maintain the relationship between Y and other covariates W as they are, and also maintain the relationship between X1 and W as they are. Perturbing the response would distort the original relationship between Y and W. The Freedman-Lane approach is computationally straightforward and solves this problem. | |
| Oct 30 at 17:23 | comment | added | jarbet | @Kozolovska: Yup. In my experience, the price you pay is the test is more conservative compared to a conditional test, although the article I linked shows the approaches have asymptotically equivalent power. The benefit of this approach is you only need to permute Y, and then from those perms you can test each feature in the model. In contrast, the conditional permutation methods, require separate rounds of permutations for each feature in the model, thus higher computational cost. If you only need 1 pvalue, this is no problem, but often I want pvalues for all features int he model. | |
| Oct 30 at 16:39 | comment | added | Kozolovska | when permuting the outcome, you are testing if the coefficient against the global null, not against the conditional hypothesis (i.e., is the coefficient not zero given all other features). | |
| S Oct 29 at 18:29 | review | First answers | |||
| Oct 29 at 23:00 | |||||
| S Oct 29 at 18:29 | history | answered | jarbet | CC BY-SA 4.0 |