How to compute p-value from a permutation test? I'm running a series of permutation tests inside 2 for loops (2 for loops will calculate some new data and then  this data will be shuffled and then i apply a linear regression based on the shuffled data). Here is my code:
for(i in seq(1,12000,200))
{
    for j in seq(1, 12000, 200))
       {
            for(ind in 1:nrow(df))
            {
            ###calculate something (called X) depending on i,j, ind and df
            }
      for (i in 1:100) # 100 sample
        {
                sample(df$X,nrow(df), replace=FALSE) 
                 fit=lm(X ~ Y, df)   #linear regression calculation
                 regerror=sum(residuals(fit)^2)
             res <- rbind(res, data.frame(shufflederr = regerror))
                 }
        }
}

The p-value is calculated based on the sum squared error of the linear regression in R which is:
> sum(res$shufflederr < observed$regerror)    #observed df contain the same error value but without shuffling

I have 3 questions:


*

*Can a p-value of a pair (i,j) be equal to 0 ??is this statistically possible ?

*Is there any optimal way to create a good cutoff of the p-value ?

*Is there any better idea to do the permutation(or any other significance) test in my case ?

 A: For number 1 the answer is no when computed correctly (but is possible when the original value is left out).  Instead of sum(res$shufflederr < observed$regerror) you should use something like: sum(c(res$shufflederr,observed$regerror) <= observed$regerror) which includes the original ordering as one of the possible permutations and switches from less than to less than or equal to, since there could be many permutations that give the same test statistic and those need to be counted as well.
For number 2, when you need a cutoff for the p-value then you should choose that cutoff in similar ways as you would for parametric or other tests.  This should represent your understanding of the consequenses of the possible errors, not something specific to the way you calculate the p-values.
For 3, look into parallel processing.  This is an example of an embarassingly parallel problem and would be very easy to parallelize.
For the innermost loop you could replace it with a call to replicate, then you don't need to keep using rbind.  If not then it is better to preallocate a matrix of the correct size and assign into the rows of that matrix than to keep rbinding to a matrix, this slows things down as you keep making copies and it fragments your memory.
