Compare two distributions of large sizes and unequal variances where one distribution is heavily skewed

Question

My data is from cells that are treated under two different conditions and then their response to the condition is measured by one output variable.

The cell populations in the two conditions are quite different, one has about 7 000 cells and one almost 30 000 cells. When I plot the histograms of the single output variable, one is heavily skewed with a long tail to the right and one is less skewed, but does not look normal. The histogram peaks are clearly separated and together with their different shapes, this make me inclined to think that there are different effects of the different conditions (biologically speaking).

However, I am having troubles expressing myself statistically. My current approach is to show the distributions in either overlapping histograms or a violin plot rather than a bar plot since I want to emphasize the difference in distribution shape as well as shift in means. I would like to accompany this with a statistical indicator of the difference between these two distributions.

When I perform a T-test or Welch test, I get super significant p-values of at least 10^-100, which, if I understand correctly, can largely be attributed to my high power from having big sample sizes and are not really indicative of a meaningful difference between the data. I am currently thinking of including the effect size of the difference in means together with this p-value and a distribution/violin plot as my final way of presenting this data. But before that, I wanted to ask here if my approach is sound or if there is a better way to show that these distributions are different from each other.

(I am sorry if there statistical language is not correct there, please ask if you need clarification of something. Many approaches I have seen to similar problems, employ a bar plot of means +-s.d. and then follow up with a T-test. I wanted to see if there is a more informative approach since feel that a lot of information is hidden by presenting it this way, but I understand it is preferred by many for practical reasons.)

Answer 1

I would not cling on to the fancy statistical treatment of your results. If you see that the distributions are very different, that alone is probably an interesting finding. You can run two-sample Kolmogorov-Smirnov test just to have a statistical analysis to support your observation, which can be shown as histograms. This is the test with null hypothesis that both samples are from the same distribution. I wouldn't focus on its results though, but simply list it as note. The graph should be enough.

Even if the outcomes had the same mean, showing that their distributions are very different can be valuable in some cases. In your case the distributions are clearly separated, so there's no need to make your results "cooler" by an array of statistical tests.

Answer 2

With such enormous sample sizes almost any small difference in distributions would give a very small p-value for testing equality of means, so it is really more interesting to describe the difference between distributions. Maybe with confidence intervals. (But with sample sizes 7000 and 30000, there must be some substructure to the data, so I would worry about possible dependencies. But no more can be said about that since we know nothing here about such substructure, so in the rest I will ignore it.)

You say: "... or if there is a better way to show that these distributions are different from each other." so I will indicate one other way which could be enlightening. One way of comparing the two distributions is simply a QQplot of one against the other, but I will indicate one other method here which could be easier to explain to the untrained eye.

I will follow the book Handcock & Morris: "Relative Distribution methods in the Social Sciences". Call one of your populations the Reference population" and the other the "comparison population", we will express the comparison distribution relative to the comparison distribution. Let the observations from the reference population be $Y_{01}, Y_{02}, \dots, Y_{0m}$ and from the comparison population $Y_1, Y_2, \dots, Y_n$. The CDF (cumulative distribution function) of the reference population $F_0$, of the comparison population $F$. Let $Y_0$ be a generic random variable with distribution $F_0$, $Y$ a generic random variable with distribution $F$. We know that $F_0(Y_0)$ is uniformly distributed. If the two distributions are equal, then also $F_0(Y)$ will be uniform. In general we say that the distribution of $F_0(Y)$ is the relative distribution of the comparison population as compared to the reference population. Now, plots of that distribution, like histograms, is a good way to express comparison of the two distributions.

If the reference distribution $F_0$ is known, we can define as **relative data* $$ F_0(Y_1), F_0(Y_2),\dots,F_0(Y_n). $$ In the more realistic case where $F_0$ is not known apriori, substitute in the above the empirical distribution function $\hat{F_0}$ for $F_0$.

Now you can make histograms and other distribution plots such as kernel density estimators , using the relative data.

One simple possibility is a QQplot of the relative data against the uniform distribution ... For some plots of relative distributions, see What are good data visualization techniques to compare distributions?.

Answer 3

A couple of points:

• If your data are not normally distributed, a t-test is probably not appropriate. You could think about transformations of the data to make it more normal (though you have to be able to interpret those transformations) - for example, logging a skewed variable can sometimes make it normal.

• You are correct that you should look at effect size not p values with that much data.

• Now the big one: Are you interested in moments or summary stats of the distributions (like means, standard deviations, medians)? Or are you interested in whether you can consider the distributions themselves the same? This has been addressed in another thread that might prove useful to you: Assessing the significance of differences in distributions