# Stacking layers of permutation: Calculating Monte Carlo p values using other Monte Carlo p values

I want to perform One-way analysis with permutation test. In this link, I found some instructions how to proceed in R. In that tutorial, the example data set has equal sample sizes by factor:

I was wondering to apply similar approach with unequal sample sizes by factors:

I believe that in this situation, some sort of control of differences in sampling effort has to be accounted, avoiding run the analysis with unequal samples sizes. Also I'm think is necessary avoid throwing data away and loss information.

To deal with this problems, I wish to use a kind of rarefaction** approach to accomplish this task, equating the effort by factors without loosing information. To do so, one could apply multiple runs of analysis, as I will try to describe below:

In the unequal example the factor A has only five observations, so, in RUN1 we rarefy all the other factors to only five random selected observations*** too, equating efforts:

Note a pseud.p value**** for RUN1. This value could be calculated with independence_test() function from package coin, using permutation procedures from the tutorial cited before. The next step is repeat the rarefaction of observations from factors B and C, selecting five observations by factor and calculating another pseudo p value, from RUN2 to RUN1000:

In the end, I wish to calculate one mean using all 1000 calculated pseudo p values. Since this procedure is repeated many many times, the information from all observations contributes to the final p value. This unified p value will be used to make inferences.

If this unified p value be significant, I wish also to apply the same rarefaction procedure on a post-hoc test using the headtail function from FSA package. See the tutorial cited before.

Now the questions:

1-Since both functions (independence_test(),headtail()) already do permutation, is correct to do what I described above? In other words is correct to add another/s "layer/s" of permutation procedures?

2-If question 1 is correct, does someone knows any paper or book that apply identical or similar approach?

** rarefaction is a procedure widely used in biodiversity analysis to deal with distinct numbers of sampling units, individuals or frequencies between factors/treatments.link

*** Note that all five observations values from factor B and C are presented on the original data-set, on which B has ten observations and C has fifteen.

**** p values only for illustration, not actually calculated

• I suspect the tests of location in the coin package aren't bothered by unequal sample sizes among groups. For example, see Coin::LocationTests, perhaps in particular the oneway_test example. Commented Sep 25, 2017 at 12:29

I don't see a reason why in this case with unequal sample sizes a different permutation approach is required. The null hypothesis is that the distribution of the Response is the same no matter whether the Factor assumes value A or B or C. Thus, if the assignment of the label A/B/C were random it can also be re-randomized without loss of information. Therefore, to obtain a $p$-value, you simply permute (many times) the observed Response on the Factor and re-compute the test statistic of your choice. This is what independence_test() does internally. Note that the function oneway_test() - as already mentioned in the comments - is a convenience interface exactly for the $k$-sample situation.
• An example with independence_test() for a categorical response (rather than oneway_test() with a numeric response) is provided in the main journal paper ("A Lego System for Conditional inference", The American Statistician, 60(3), 257-263. Vignette version: cran.r-project.org/web/packages/coin/vignettes/LegoCondInf.pdf). See the section "Contingency Tables: Smoking and Alzheimer’s Disease". The function oneway_test() can be used in the same way for a numeric response. Commented Oct 24, 2017 at 20:21