# Which statistical significance test to use?

I have got the following population within a data.frame:

df = read.table(text = 'ID  Value
1   30
2   10
3   13
4   32
5   22
6   45
7   12
8   9
9   1
10   6
11   56
12   7
13   28


Now I will divide the above data.frame in two: df1 with Value >= 20 and df2 with Value < 20.

df1 = read.table(text = 'ID    Value
1     30
4     32
5     22
6     45
11     56

df2 = read.table(text = 'ID   Value
2    10
3    13
7    12
8     9
9     1
10     6


And I will calculate the two relative means:

mean(df1$Value) = 35.5 mean(df2$Value) = 8.3


Now my question: In order to test the significance of these two mean values, which statistical test should I apply and how?

• This question is off topic for this site. Feb 11 '17 at 21:55
• I don't see any reason to believe this question is off topic here, @MichaelChernick. The question includes R code, but it doesn't ask for code--it asks what test to use. It would be off topic on Stack Overflow, however. I'm voting to leave open. Feb 12 '17 at 2:30
• I really don't see how including the data makes the statistical question being asked off topic. Feb 12 '17 at 4:21
• This is like asking "do people that earn over \$20,000 earn more on average than people that earn below \$20,000?" ... the answer is obvious (of course they do!) and it doesn't seem to be a statistical question at all. Please explain WHY you're doing this (what you're trying to do that led you to propose this approach), because your proposed approach causes answers that take the question at face value to be nonsensical. Feb 12 '17 at 4:25
• @aaaaa For example what's the actual research question that you are thinking about here? (Note that if you split like this, the means of the two corresponding truncated populations are different by construction, so how can you be testing for equality of means? You know before you start that they can't be the same) Feb 12 '17 at 7:09

Every statistical test you use normally has some sort of assumptions about the underlying data. Since you have not stated any, I'll assume you want a test which works without making such assumptions (ie. Normality or equal variance).

One such test would be a randomisation or permutation test.

Suppose our null hypothesis is this:

$H_0: \mu_1 = \mu_2$

And we have a bunch of two dimensional data points: One dimension specifies their value, and the other specifies whether they belong to group 1 with mean $\mu_1$ or group 2 with mean $\mu_2$.

Our test statistic will be the difference $\bar{x}_1-\bar{x}_2$, that is the difference between the sample means of both groups. Now, if our null hypothesis is true then reassigning our values to different groups should not change this test statistic significantly, since each group has the same mean anyway. So one way to perform a statistical test is as follows:

1. Calculate the test statistic $\bar{x}_1-\bar{x}_2$
2. For i in 1,2,3,...,I Permute the collumn of group values so that each value has been assigned a new group. Recalculate the test statistic and denote it by $\bar{x_i^*}$
3. Compare the original test statistic to your distribution of $\bar{x_i^*}$'s.

If the null hypothesis is true, the original test statistic should be from this distribution and so should not be an 'extreme' value. If your test statistic is far into the 'tails' of the distribution you reject the null hypothesis.

More formally, if the absolute value of your test statistic is greater than the absolute value of 95% of your generated $\bar{x_i^*}$'s, then at the 5% level you consider it extreme and reject the null hypothesis.

• I would just clarify should sampling with or without replacement be performed. To my taste with replacement is more conservative and, thus better.
– zlon
Feb 11 '17 at 22:21
• Given that the original question includes the step of dividing the data into groups with values above and below 20 there is no purpose in any test of significance. There is a difference between the groups that has been imposed by the procedure and so no statistical test is appropriate. (Other than that, your advice is good.) Feb 12 '17 at 7:35

Your data looks strange, but nevertheless:

Step1 Let's check our data for normality

df1.shapiro=shapiro.test(df1$Value) df2.shapiro=shapiro.test(df2$Value)


Both subsets are normal (p-value > 0.1)

** Step 2 Let's check equality of variances

var.test(df1$Value, df2$Value)


Variances are not equal

**Step 3 Perform t-test

t.test(df1$Value,df2$Value, var.equal=FALSE, paired=FALSE)


p-value<0.05. So, these groups have different means.

It is better not to do Step 1 and 2, and use some a priory knowledge about Your data instead. But in these task You did not provide any information.

Be careful If tests for normality and variance equality will give you another answers You should use other tests. This is a solution for Your numbers, but not some universal recipe.

• I'm sorry but this answer is poor advice on several grounds. For starters, if you split your data into groups based on the response itself, the t-test will NOT have the required properties. Feb 12 '17 at 4:24
• I wrote that it is strange. But let's start from the apaoint we have 2 groups already
– zlon
Feb 12 '17 at 6:31
• I don't think you understand the depth of the problem here. Feb 12 '17 at 7:06
• You (and me also) don't like that he want to compare means of presorted data. OK. But if he will not write this part their is a solution. Is it allowed to me to compare weight of mice and elephant s?
– zlon
Feb 12 '17 at 7:09
• @zion You can compare the weights of mice and elephants if you choose to, but no statistical procedure is useful for comparing the weights of heavy elephants and light elephants: they are different by definition. Feb 12 '17 at 7:38