What statistical test of significance should I use when comparing two small, unequal groups of non-normal distribution?

Question

I am trying to analyze quantitative data between two independent groups. One group has 8 data points, and the other has 9. I used a Shapiro-Wilk test for each of the groups to determine normality, and one reflected normal distribution, while the other did not. Would a Mann-Whitney U test be the correct approach to analyze this data, or is there another, more appropriate test?

Answer 1

I would suggest that if you are concerned about violating assumptions of the standardized tests, you can try to run a simulation. For example, imagine shuffling the data. You put all of your 17 values in one bucket, then you randomly draw 8 from the bucket and that becomes your temporary group 0 (the rest are in group 1). Find the difference of the means. Then repeat this process many times.

Once you have done this, you can compare your observed difference of the means to the number of simulations that end up with values as larger (or small or extreme) as your original value.

Here is a bit of code to demonstrate the process. (When you run this, I suggest changing the n.sim value to 10^4 or higher.

set.seed(317) # this is not needed, just 
    # provided here so this demonstration 
    # simulation is consistent
dv.1 <- rnorm(8) # replace your first group here
dv.2 <- rnorm(9) # replace your second group here
grp <- c(rep.int(0,8),rep.int(1,9)) 
     # creates a dichotomous grouping variable
data.fr <- data.frame(grp, "dv"=c(dv.1,dv.2)) # combines the data into one bivariate data frame

# this is the test statistic:  the difference in 
# the group means M_1 - M_0
test.stat <- diff(aggregate(dv ~ grp, data=data.fr, mean)[,2])

n.sim <- 10^3 # change this to say 10^4 or higher for the final simulation
M.diffs <- rep.int(NA,n.sim) # creates a vector to store the simulate mean differences
tmp.df <- data.fr # a copy of your original data that can be shuffled
for(ctr in 1:n.sim) {
    tmp.df$grp <- sample(tmp.df$grp) 
    # shuffle the grouping variable
    M.diffs[ctr] <- diff(aggregate(dv ~ grp, data=tmp.df, mean)[,2])  # M_1 - M_0
}

# left-tailed P-value estimate
length(which(M.diffs <= test.stat))/n.sim

# right-tailed P-value estimate
length(which(M.diffs >= test.stat))/n.sim

# 2-tailed P-value estimate
length(which(abs(M.diffs) >= abs(test.stat)))/n.sim

The advantage to this approach is that it has no assumptions that need to be met.