I've just received peer reviews for an applied paper that reported a table of multiple linear regression coefficients. While I reported 95% confidence intervals for these coefficients, a reviewer has requested I report exact p-values as well. The difficulty is that the confidence intervals were estimated via an ordinary non-parametric bootstrap, given the peculiarities of the data set.

Is there a way I can estimate 'pseudo p-values' from these confidence intervals? If so, under what assumptions/conditions are these p-values reasonable?


For bootstrapping, I'm assuming you resampled with replacement, computing the beta value each time to generate a distribution from which you computed the 2.5%ile and 97.5%ile. This gave you 95% confidence intervals.

For permutation, you'd resample from without replacement (shuffling up the case/control status or whatever the main factor is), get a distribution of null p values, and then compute you empirical p value based on wherever your observed falls in this null distribution. For details on the math, check out the book recommended on the link I posted above. Sorry for the piecemeal reply.

  • $\begingroup$ Thanks, this is helpful. Is it reasonable to report both p-values calculated from a permutation test and confidence intervals calculated by a non-parametric bootstrap? $\endgroup$
    – fmark
    Apr 28 '15 at 6:06
  • $\begingroup$ Another obvious question (I'm new to permutation tests): Would I "get a distribution of null p values" or get a distribution of beta coefficients under the null hypothesis of no relationship? $\endgroup$
    – fmark
    Apr 28 '15 at 6:10
  • 2
    $\begingroup$ Please don't post one-line answers (your other one); they're generally better suited to comments. Please don't post two versions of the same answer from two different accounts. Please delete the shorter one. Please arrange to merge your accounts. $\endgroup$
    – Glen_b
    Apr 28 '15 at 6:19
  • $\begingroup$ You can report both the permuted p value and the 95% CI. In fact, I'd say it's always good to report both when possible. By the way, if your regression analysis involves time series or repeated measures, the permutation process can get a little bit more complicated than simple shuffling. Please see the following posts: stats.stackexchange.com/questions/20217/… and stats.stackexchange.com/questions/3520/… $\endgroup$
    – N P
    Apr 28 '15 at 7:01
  • $\begingroup$ @Glen_b, sorry about that. I am working on resolving the issue and I will go through the site tutorial. $\endgroup$
    – N P
    Apr 28 '15 at 7:03

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.