Can the group recommend a good introduction text/resource to applied resampling techniques? Specifically, I am interested in alternatives to classical parametric tests (e.g. t tests, ANOVA, ANCOVA) for comparing groups when assumptions such as normality are clearly violated.

An example problem type I would like to educate myself as to a better way to solve may involve something such as:

2 Groups: Treatment and Control

Dependent Var: Change in account balance dollars after intervention

Covariate: Pre intervention account balance dollars.

Issue with applying ANCOVA: Many subjects will not have any change (many zeros).

2 Groups: Treatment and Control

Dependent Var: new accounts added

Covariate: Pre intervention number of accounts.

*Many subjects will not have any added account (many zeros).

Can I use a bootstrap? A permutation test? This is the type of analysis I would like to apply nonparametric resampling methods to.


3 Answers 3


As for a good reference, I would recommend Philip Good, Resampling Methods: A Practical Guide to Data Analysis (Birkhäuser Boston, 2005, 3rd ed.) for an applied companion textbook. And here is An Annotated Bibliography for Bootstrap Resampling. Resampling methods: Concepts, Applications, and Justification also provides a good start.

There are many R packages that facilitate the use of resampling techniques:

(There are many other packages...)

  • $\begingroup$ @chi and @whuber: Thank you, I was wondering if The Good book was a good choice. For the problem types I laid out - basically ANCOVA with violations, am I on the right track with permutation or bootstrapping? $\endgroup$
    – B_Miner
    Dec 30, 2010 at 20:22
  • $\begingroup$ @user2040 It's hard to go wrong with permutation tests. Good has chapters specifically on multifactor designs, categorical data, and multivariate analysis (including MANCOVA). Although I don't fully comprehend your specific problem, I'm sure you're find something useful there. $\endgroup$
    – whuber
    Dec 30, 2010 at 22:20
  • $\begingroup$ @user2040 I will add some references but I found your two points difficult to understand as well. To my knowledge, there's no exact permutation test when the covariate is continuous. $\endgroup$
    – chl
    Dec 30, 2010 at 22:29
  • 1
    $\begingroup$ @chl I think, if I correctly understand your comment about continuous covariates, that exactness depends on the role played by randomness in the data. When randomization occurs by design, it doesn't seem to matter what kind of data you have. The permutation test takes the data as given and simply lets us glimpse how the statistical results would turn out if our random number generators had (for example) resulted in different assignments of subjects to treatment and control groups. $\endgroup$
    – whuber
    Dec 30, 2010 at 22:36
  • $\begingroup$ @chi and @whuber, Thanks again. I will see which of the Good books is the best (many puns intended). As far as my problem, basically it is a two sample experiment (treatment and control/no treatment) where there exist a pre-experiment baseline measure and a post treatment measure, the latter being the dependent variable (actually it is change in the measure from pre to post). So it would be a typical ANCOVA or ANOVA (depending on if the change is the dependent or post is, with the pre as a covariate) except that many of the post measurements are zero (the customer did not purchase anything). $\endgroup$
    – B_Miner
    Dec 31, 2010 at 0:55

Phillip Good, Permutation, Parametric, and Bootstrap Tests of Hypotheses (3rd Edition). Springer, 2005.

This book is mathematically easy, accessible, and covers a wide range of applications.

  • $\begingroup$ (+1) Ah... we cited the same book :) $\endgroup$
    – chl
    Dec 30, 2010 at 20:21
  • $\begingroup$ @chl I don't think so: they're by the same author but have slightly different titles and different publishers. Maybe we should each say a little more about them so we can determine which might be more appropriate for the OP. I added a few details in a comment to your response. $\endgroup$
    – whuber
    Dec 30, 2010 at 22:23
  • $\begingroup$ I deleted mine after seeing yours. $\endgroup$
    – chl
    Dec 30, 2010 at 22:26
  • $\begingroup$ @chl Ah, I see. So there's no redundancy. $\endgroup$
    – whuber
    Dec 30, 2010 at 22:37

Aside from the excellent and cleverly written Good's book mentioned in the response above, you might find Basso, Pesarin, Salmaso, and Solari "Permutation Tests for Stochastic Ordering and ANOVA Theory and Applications with R", useful.

To be honest, I haven't read it all the way through, but it employs some new models based on likelihood and sufficient statistics (if that makes sense in a nonparametric context). It is simple to read (for someone who knows statistics), but it contains all of the theoretical details as well as some R.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.