Best statistical test to compare two groups when they have different distributions

Question

I am teaching myself to use and apply statistics to a big database.

• I have 2 groups that I wish to compare, healty controls (HC) and patients (P).

• The sizes are HC= 84, P= 196.

• Each group has many different clinical data collected as continous variables, such as weight, BMI, size of theire frontal lobe, etc.

Given the central limit theorem, I would have thought that I could just assume normality. I corroborated this graphically with histograms, and noticed that in many cases the distribution was not normal. So I performed a shapiro-wilk test, confirming that several variables of both patients and controls had non-normal distribution.

This outcome made me more confused, because now I would like to compare the 2 groups, but they have different distributions.

How should I statistically approch this?

For example, I would like to test if the size of the frontal lobe is statistically different in patients than in controls. BUT the distributions looks like this:

Thanks!

Answer 1

From your example, I would probably be fine using a t test in that case. The data are relatively symmetrical, and not terribly skewed. As you mention, considering the Central Limit Theorem for data like this and your sample size, there's probably no problem with the data's deviations from an approximately normal distribution. The groups appear to have relatively similar variances, but you could use Welch's t test if this is a concern for some variables.

For a two sample comparison, there are lots of different tests you could use depending on what you want to compare about the samples. t test, Wilcoxon-Mann-Whitney, a median test, various permutation tests, maybe comparing a percentile other than the median if that's of interest. You might start by thinking about what you want to compare. Is it the mean? median? stochastic dominance? 95th percentile? variance?

The Shapiro-Wilk test in this case is probably not telling you anything useful. From your histograms, I can tell that your example data deviate from a normal distribution. But that doesn't necessarily inform what analysis you should perform.

Answer 2

One option would be to use a two-sample Kolmogorov-Smirnov test:

https://en.wikipedia.org/wiki/Kolmogorov%E2%80%93Smirnov_test

This compares the empirical CDFs of the distribution, and computes a test statistic based on the quantile based on the largest discrepancy between the two. The KS test does not require any assumptions about the distributions that the two samples follow. With a large enough sample this test can state whether there is an indication of them following different distributions, though it will not quantity how they are different.