2
$\begingroup$

Can you please suggest the metrics/tests for accessing the quality of Y-randomization (permutation)?

I have some doubts in quality of Y-randomization that I am using with sample.int from R. For 100 Y-randomization runs I found that in average 2% of the instances are falling into themselves, and around 5% of instances are falling into the similar instances with correlation higher then "95% quantile of correlation matrix for initial data".

The dataset is rather small ~80 instances and have some similar entries with correlation median for all against all around 0.55.

Update: part of a code I am using for modeling (RSNSS)

# t_comb_orig - original data, input only

# original data and it's correlation
t_comb_orig <- rbind(t_train_in, t_test_in)
cor_orig <- matrix(NA,nrow(t_comb_orig),nrow(t_comb_orig))
for(l in 1:nrow(t_comb_orig)) {
  for(k in 1:nrow(t_comb_orig)) {
    cor_orig[l,k] <- cor(t_comb_orig[l,],t_comb_orig[k,])
  }
}
cor_orig[lower.tri(cor_orig,diag=T)] <- NA
cor_orig_median <- median(c(cor_orig),na.rm=T)
cor_orig_095qutl <- quantile(c(cor_orig),0.95,na.rm=T)
# modeling
for (i in 1:N) {
    # randomize
    t_comb_rand <- t_comb_orig[sample.int(nrow(t_comb_orig)),]
    t_train_inR <- t_comb_rand[1:nrow(t_train),] 
    t_test_inR <- t_comb_rand[(nrow(t_train)+1):nrow(t_comb_rand),]
    # train model
    t_model <- mlp(x=t_train_inR, y=t_train_out,size=c(floor(ncol(t_train_inR)/2)), learnFunc="Rprop", inputsTest = t_test_inR, targetsTest = t_test_out, maxit=50)
    t_ext_prediction <- predict(t_model,t_ext_in)
    write.table(ifelse(t_ext_prediction >= 0.5, 1, 0), file=paste("test__",i,".txt",sep=""),sep="", col.names = F, row.names = F)
    # plot error
    png(filename=paste("test__",i,".png",sep=""))
    plotIterativeError(t_model)
    dev.off()
        # randomization test (correlation between scrambled vs real)
        for (j in 1:nrow(t_comb_orig)) scrambling_test[j,i] <- cor(t_comb_orig[j,],t_comb_rand[j,])
}
#
write.table(scrambling_test,file="scrambling_vs_real_cor.txt",sep="\t", col.names = F, row.names = F)
scrambling_test_per_run <- matrix(NA,ncol(scrambling_test),1)
for (o in 1:ncol(scrambling_test)) scrambling_test_per_run[o,1] <- length(which(scrambling_test[,o] > cor_orig_095qutl))/nrow(scrambling_test)
write.table(scrambling_test_per_run,file="scrambling_vs_real_cor_per_run.txt",sep="\t", col.names = F, row.names = F)
# 
$\endgroup$
  • $\begingroup$ Please provide some code of what you've tried. $\endgroup$ – Sven Hohenstein Feb 9 '13 at 3:45
  • $\begingroup$ Please, look at the update of the post. $\endgroup$ – Vladimir Chupakhin Feb 9 '13 at 4:17
4
$\begingroup$

Since your code isn't reproducible, I ran a simple test of the sample.int function. But I failed to replicate your results.

set.seed(42)

samMat <- replicate(10000, sample.int(80))
  # randomly shuffle the numbers 1-80 (10,000 times)

corMat <- cor(samMat) # compute correlations

summary(corMat[lower.tri(corMat)]) # summary (only the lower triangular is used)

      Min.    1st Qu.     Median       Mean    3rd Qu.       Max. 
-0.5088000 -0.0681300  0.0000000  0.0000035  0.0681300  0.5913000

As you can see, both median and mean of the correlation coefficients are zero. The absolute maximum is below $|r| < .60$. Here is a histogram of the results:

enter image description here

As you can see, the sample.int functions works as it is supposed to do. In 10,000 runs I never obtained identical datasets. Of course, the correlations are influenced by the number of values in the dataset. Is the return value of nrow(t_comb_orig)) really 80? If this number is lower, there are less possible permutations and thereby stronger correlations. Note: the maximum number of permutations of a vector of length $n$ is $n!$.

In your code, you use the random numbers for indexing t_comb_orig. If this object contains duplicated values, the correlations will inherently be higher.

You should check whether your code works correctly. There's no error in the sample.int function.

$\endgroup$
  • 1
    $\begingroup$ thanks you! the data is sparse and do have high degree of self-correlation with median around 0.55.T here is open question for me how to measure the quality of randomization for such data. Is correlation enough? if we have the some degree of self correlation how to treat the prediction of the Y-scrambled data? If we have R2 of normal data is 0.85 and Y-randomized 0.55 (median) is it good or bad? where is the threshold? $\endgroup$ – Vladimir Chupakhin Feb 9 '13 at 23:47
  • $\begingroup$ @VladimirChupakhin The answer to your question depends on (a) the length of the vector (which is shuffeld), (b) whether there are duplicated values, and (c) the number of samples. The probability masters on this site should be able to easily answer your question. But you should provide more information. $\endgroup$ – Sven Hohenstein Feb 10 '13 at 8:26
  • $\begingroup$ thank you, I will rephrase and reask the question. I found some literature on this question, studying it. $\endgroup$ – Vladimir Chupakhin Feb 12 '13 at 17:04

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.