# Comparing Result of Permutation and bootstrapping

I am trying to verify the correlation between two parameters using bootstrap/permutation methods(classical example!). What i understand is that both permutation and bootstrap method involves shuffling the relationshiop between the two variables repeatedly and recalculating the correlation. However, histogram of Matlab function and my version of bootstrap give very different figures

using Matlab function to verify correlation:

load lawdata  % this is matlab data example
[rhohat phat] = corr(lsat,gpa);
rng default  % For reproducibility
rhos1000 = bootstrp(1000,'corr',lsat,gpa);
figure,hist(rhos1000,30,'FaceColor',[.8 .8 1])


Nearly all the estimates lie on the interval [0.4 1.0].

but if i use the following code

x1=lsat; n1=length(x1);
x2=gpa;  n2=length(x2);
myStatistic = @(x1,x2) corr(x1,x2);
sampStat = myStatistic(x1,x2);
mybootstrap = zeros(nReps,1);
for i=1:1000
sampX1 = x1(ceil(rand(n1,1)*n1));
sampX2 = x2(ceil(rand(n2,1)*n2));
mybootstrap(i) = myStatistic(sampX1,sampX2);
end

xx = min(mybootstrap):.01:max(mybootstrap);
hist(mybootstrap,xx);


histogram plot looks like a normal distribution with mean around zero

(1)- Would you please kindly help me to figure out where is my mistake or misunderstanding? where the difference is coming from?

(2) - i think bootstrap is used to estimate the confidence intervals and the permutation test estimates significant corrlation, here, p = sum(mybootstrap>sampStat)/nReps; . should we report both results for our statistical analysis?

many thanks Karlo

The two methods address different problems.

The bootstrap is for estimating the sampling distribution of your statistic. You repeatedly draw samples (with replacement) from your data and calculate the correlation for each sample. Loosely, the bootstrap is trying to answer the following question: if you were to repeat the experiment many times, what would the distribution of the correlation look like over these repetitions?

The permutation procedure is for estimating the null distribution of your statistic. Permutation testing destroys the correlation by randomly shuffling the data. Loosely, it's trying to answer the following question: If there were truly zero correlation, what would the distribution of the correlation look like across many runs of the same experiment?

The bootstrap is useful for calculating confidence intervals, whereas permutation tests are useful for testing the null hypothesis of zero correlation (i.e. calculating p values).

Since you're using Matlab, you can use the function bootci() to calculate bootstrap confidence intervals. By default, it uses the 'bias-corrected, accelerated bootstrap', which is an improvement over simply taking percentiles of the bootstrap distribution as the confidence interval.

• Thank you @user20160 so much for your help and clarifying the confusion. – Karlo Gonzales Jul 25 '16 at 14:17

Thank you @user20160 so much for your help and clarifying the confusion. I modified the permutation code as follow and got the same results as bootstrap function in matlab, hope it helps others as well!

x1=lsat; n1=length(x1);
x2=gpa;  n2=length(x2);
myStatistic = @(x1,x2) corr(x1,x2);
sampStat = myStatistic(x1,x2);
mybootstrap = zeros(1000,1);
for i=1:1000
rd=ceil(rand(n1,1)*n1)
sampX1 = x1(rd);
sampX2 = x2(rd);
mybootstrap(i) = myStatistic(sampX1,sampX2);
end
xx = min(mybootstrap):.01:max(mybootstrap);
figure,hist(mybootstrap,xx);