Can I use permutation tests for to avoid the multiple comparison problem in the context of proportions? I am evaluating the effectiveness of 5 different methods to predict a particular binary outcome (call them 'Success' and 'Failure'). The data look like so:
Method    Sample_Size    Success    Percent_Success
1         28             4          0.14  
2         19             4          0.21  
3         24             7          0.29  
4         21             13         0.61  
5         22             9          0.40 

I would like to perform a test among these 5 methods to assess the relative superiority of the methods. In other words, I want to order the methods in order of performance as method 1 > method 2 > ... method 5. To avoid the issue of multiple comparisons, I plan to do a permutation test along the following lines:
Step 1: Pool all the data so that the overall sample size is 114 with overall 37 successes. 
Step 2: Randomly split the data into 5 groups with the corresponding sample sizes of 28, 19, 24, 21 and 22.
Step 3: Increment a counter if the observed order of Percent_Success from step 2 is consistent with the ordering of my data.
Step 4: Repeat steps 2 and 3 many times (say 10000).
Desired p-value = Final Counter Value / 10000.
Questions:


*

*Is the above procedure ok?

*Is there anything in R that would enable me to perform the above test?

*Any suggestions for improvement or alternative methods would be helpful.  
 A: The proposed procedure does not answer your question.  It only estimates the frequency, under the null hypothesis, with which your observed order would occur.  But under that null, to a good approximation, all orders are equally likely, whence your calculation will produce a value close to 1/5! = about 0.83%.  That tells us nothing.
One more obvious observation: the order, based on your data, is 4 > 5 > 3 > 2 > 1.  Your estimates of their relative superiorities are 0.61 - 0.40 = 21%, 0.40 - 0.21 = 11%, etc.
Now, suppose your question concerns the extent to which any of the ${5 \choose 2} = 10$ differences in proportions could be due to chance under the null hypothesis of no difference.  You can indeed evaluate these ten questions with a permutation test.  However, in each iteration you need to track ten indicators of relative difference in proportion, not one global indicator of the total order.
For your data, a simulation with 100,000 iterations gives the results
\begin{array}{ccccc}
  & 5 & 4 & 3 & 2 \cr
 1 & 0.02439 & 0.0003 & 0.13233 & 0.29961 \cr
 2 & 0.09763 & 0.00374 & 0.29222 &  \cr
 3 & 0.20253 & 0.00884 &  & \cr
 4 & 0.08702 &  &  & 
\end{array}
The differences in proportions between method 4 and methods 1, 2, and 3 are unlikely to be due to chance (with estimated probabilities 0.03%, 0.37%, 0.88%, respectively) but the other differences might be.  There is some evidence (p = 2.44%) of a difference between methods 1 and 5.  Thus it appears you can have confidence that the differences in proportions involved in the relationships 4 > 3, 4 > 2, and 4 > 1 are all positive, and most likely so is the difference in 5 > 1.
A: Your suggested Monte-Carlo permutation test procedure will produce a p-value for a test of the null hypothesis that the probability of success is the same for all methods. But there's little reason for doing a Monte Carlo permutation test here when the corresponding exact permutation test is perfectly feasible. That's Fisher's exact test (well, some people reserve that name for 2x2 tables, in which case it's a conditional exact test). I've just typed your data into Stata and -tabi ..., exact- gave p=.0067 (for comparison, Pearson's chi-squared test gives p=.0059). I'm sure there's an equivalent function in R which the R gurus will soon add.
If you really want to look at ranking you may be best using a Bayesian approach, as it can give a simple interpretation as the probability that each method is truly the best, second best, third best, ... . That comes at the price of requiring you to put priors on your probabilities, of course. The maximum likelihood estimate of the ranks is simply the observed ordering, but it's difficult to quantify the uncertainty in the ranking in a frequentist framework in a way that can be easily interpreted, as far as i'm aware.
I realise I haven't mentioned multiple comparisons, but I just don't see how that comes into this.
