I'm not strong in statistics and I'm looking for a help. I work with real estate data and I want to compare apartment prices in 2 districts: district "A" and district "B".

Data description:

array "A":

$mean_A$ = 368.52

$std_A$ = 256.02

$n_A$ = 2040

array "B":

$mean_B$ = 340.02

$std_B$ = 284.02

$n_B$ = 2741

Looking at the data, it seems that district A is more expensive than district B. I want to be sure about it and make an experiment.

The observed statistic : $mean_A - mean_B$ = 28.5

$H_0$: $mean_A - mean_B$ = 0

$H_a$: $mean_A - mean_B$ > 0

That I did:

  1. I shifted array A by 28.5(observed statistic) to center $mean_A - mean_B$ right on 0.

  2. I simulated 10000 random samples for array A and for array B (sampling with replacement), get 10000 mean-differences and save them.

  3. Next I want to see there the original observation(28.5) is on that distribution mean differences and calculate p-value.

p-value = (the number of values that are >= 28.5 ) / 10000

and is equal to 0.001.

So the difference is significant and I can reject $H_0$.

Tell me please, is it ok this procedure or I'commited mistake.

I would appreciate any piece of advice. Thank you very much for your time and effort.


What you are doing seems more like a bootstrap procedure than a permutation test. Because I don't have your data or a description of it, I can't be sure why you aren't using a Welch 2-sample t test or a 2-sample Wilcoxon rank sum test.

Suppose I have data as below, as sampled in R. Even though sample sizes are very large, I would not want to trust a t test with such severely skewed data.

summary(a);  length(a);  sd(a)
   Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
  112.2   188.8   285.1   368.7   463.2  1904.2 
[1] 2040
[1] 254.4081
summary(b);  length(b);  sd(b)
   Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
  56.03  141.93  256.71  347.55  461.82 2611.67 
[1] 2741
[1] 292.5565
x = c(a,b);  gp = rep(1:2, c(2040,2741))
boxplot(x ~ gp, col="skyblue2", pch=20)

enter image description here

I will not use a t test because I do not trust the t statistic to have t distribution. However, I will use the pooled 2-sample t test statistic as my 'metric' for a permutation test, because I feel the t statistic is a reasonable way to express the difference between values in groups A and B.

t.obs = t.test(x ~ g, var.eq=T)$stat;  t.obs

Now I will use sample(gp) to randomly permute the 1s and 2s in gp. All $2040 + 2741$ are reassigned to groups 1 and 2 with $n_1 = 2040$ in group 1 and the rest in group 2. On a much smaller scale the code below illustrates one permutation of the vector c(1,1,1,2,2,2,2).

[1] 2 2 2 1 1 2 1

I will do $10\,000$ permutations of gp, finding t.prm for each permutation. Then the P-value of the permutation test will be the proportion of the values in t.prm that are larger in absolute value than t.obs for the original unpermuted data.

t.prm = replicate(10^4, t.test(x~sample(gp),var.eq=T)$stat)
mean(abs(t.prm) > abs(t.obs))
[1] 0.0079

The P-value is about $0.008$ so we reject the null hypothesis that the two groups have the same mean.

enter image description here

hist(t.prm, prob=T, br=30, col="skyblue2", 
     main="Simulated Permutation Dist'n")
  abline(v = c(-t.obs, t.obs), col="red", lwd=2, lty="dotted")

Notes: (1) A Wilcoxon rank sum test shows that the two groups have significantly different locations:

wilcox.test(x ~ gp)

        Wilcoxon rank sum test with continuity correction

data:  x by gp
W = 3138800, p-value = 3.737e-13
alternative hypothesis: true location shift is not equal to 0

(2) For a one-sided test, as in your question, the last line of R code for the P-value would be as follows:

mean(t.prm > t.obs)
[1] 0.0042

(3) I am not familiar with the test you did. It seems to be a bootstrap test (because of re-sampling with replacement). I'm not saying your test is wrong; it gives a reasonable result. However, you asked about a permutation test; your test is not a permutation test; so I showed you one.

(4) The R code below was used to obtain the two samples used in the discussion above:

a = rexp(2040, 1/256) + 112
b = rexp(2741, 1/284) + 56
x = c(a,b);  gp = rep(1:2, c(2040,2741)
  • $\begingroup$ Thank you very much for your answer, as I said I'm not strong in Statistics. I'm not familiar with Welch o Wilcohox test, neither familiar with R(only Python). I'm going to read about this two tests and read about the difference between bootstrap test and permutation test. $\endgroup$ – audiotec Sep 14 '20 at 12:13
  • $\begingroup$ If you have specific questions about the R code, please ask. I will try to explain the most mysterious parts. At the end: t.perm is a vector of $m=10\,000$ t statistics from permuted tests; abs(t.prm) > abs(t.obs) is a logical vector of $m$ TRUEs and FALSEs and its mean is the proportion of its TRUEs. // The Welch t test requires nearly normal data, but does not require that the two group variances be equal. The Wilcoxon rank sum test does not require normal data, but works most transparently if the two samples are from populations of about the same shape. $\endgroup$ – BruceET Sep 14 '20 at 17:56
  • $\begingroup$ Thank you, very much for your help. You were right this is Bootstrap procedure I was reading some papers and I found an error in my procedure: 3. Next I want to see there the original observation(28.5) is on that distribution mean differences and calculate p-value. <= here I made a mistake. The right way to shift my data is: array(A) = array(A) - mean(A) + mean(of both arrays) array(B) = array(B) - mean(B) + mean(of both arrays) Only then I can generate random data and find p-values. Sorry for my question "off-topic" it because of my ignorance in the subject. $\endgroup$ – audiotec Sep 15 '20 at 18:37
  • $\begingroup$ Glad you've resolved this. $\endgroup$ – BruceET Sep 15 '20 at 18:47

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.