Skip to main content
10 of 14
Grammar
NotMe
  • 761
  • 1
  • 6
  • 19

Hypothesis test: numeric vs. ranked

I believed that the most powerful hypothesis test for judging whether a single sample comes from $N(0,1)$ or from $N(1,1)$ uses the average value as test statistics. Thus, I calculate the sample size using a Monte Carlo simulation and compared two methods: (a) using the numeric average value as test statistic, and (b) transforming the data first to ranks and then calculating the average value. What I found is that the ranked data yields a smaller sample size. How is this possible?

What I am finally interested in is to determine the sample size for a future experiment. I like to take $n$ samples from an industrial process with a known distribution, analyse the samples and show that the process is well within its specification limits. This allows me to produce the product without measuring each part.

There are many subtle difficulties involved in the hypothesis test. That's why I setup the following toy model to focus the attention on the above described "numeric vs. ranked" question.

set.seed(2020)
transformToRank = FALSE
alpha           = 0.05
betaTarget      = 0.20

T = matrix(nrow = 1e4, ncol = 2) # alloc space for test statistic
for ( nSample in 3:100 ) {
    for ( i in 1:1e4 ){
        # combine data:
        random = c(rnorm(nSample,0,1), rnorm(nSample,1,1))
        if ( transformToRank ){
            # rank data (combined):
            random = rank(random)
        } 
        # Calc test statistic:
        T[i,] = c(mean(random[1:nSample]), mean(random[(nSample+1):(2*nSample)]) );
    }
    cutoff1 = quantile(T[, 1], alpha/2)
    cutoff2 = quantile(T[, 1], 1-alpha/2)
    beta    = sum(cutoff1 < T[, 2] & T[, 2] < cutoff2) / 1e4
    if ( beta <= betaTarget ){
        break 
    } else {
        nSample = nSample + 1
    }
}
print(beta)
print(nSample)

If I use the numeric value, the sample size $n=8$ satisfies the $\beta$-risk condition. In contrast, if I transform the random numbers to ranks, I only need $n=4$. Thus, if I am willing to use a randomly generated dataset to analyse (rank) the experimental data, the power of the test increases significantly. This logic also applies if I sample from a location scaled version of the $t$-distribution. What am I missing?

NotMe
  • 761
  • 1
  • 6
  • 19