# Hypothesis testing for multiple distributions

I have a set of distributions (histograms) relating to data sampled from two different populations. Within each population I have $n=15$ distributions. I am looking for a statistical test that will tell me whether the two populations are statistically different, or not, based on the distributions within them.

Most tests I have looked at (e.g. Mann-Whitney U, Wilcoxon) seem to compare two samples. However, since I have 15 samples in each population, I am confused as to whether these tests can be applied in this instance. Is it possible to just pool the data so that I have one aggregated distribution for each population, then run a test on the result?

EDIT

Here is a selection of 5 distributions (histograms) from each population, plotted on the same axes:

I want to know if there is a statistical difference between the sets (black and orange) of distributions.

-
Are you saying that you have measured 15 different variables (yielding 15 distributions) in two different populations, and you are looking for "overall" difference? –  fileunderwater Nov 20 '13 at 10:25
@fileunderwater I have measured the same variables, but for 15 different individuals within each population. I'd like to know whether the populations are "different" in a statistical sense. –  allhands Nov 20 '13 at 10:26
@allhands Ok, so you have 15 datapoints for each measured variable (taken from 15 individuals). How many variables have you measured in each population? –  fileunderwater Nov 20 '13 at 10:29
@fileunderwater For each individual there is a histogram which maps the distribution of a single measured variable. The same variable has been measured for all individuals. There is one distribution per individual, so 15 distributions per population (30 in total). –  allhands Nov 20 '13 at 10:53
@allhands Maybe I'm just thick, but to me this is still unclear. Have you measured one single variable multiple times for each individual (e.g. as a time series)? How many datapoints do you have in total (all datapoints used in histograms over all individuals)? Either way, your question would be more clear if you could include a sample dataset, to show how the data is structured. –  fileunderwater Nov 20 '13 at 11:19

For a simple test of the difference between populations I would consider using the mean for each individual, and a Mann-Whitney U or t-test to test for a difference between the 15 individuals in each population (since you are only comparing a single variable). You could also do glm where you include 'individual' as a random effect, and test for the fixed effect of 'population'. However, when looking at your plot, it looks like you are mostly interested in the difference in variance between populations. Have the measurements been standardized within individuals or do all hover around zero as raw data? To test for a difference in variance between populations (using means of individuals) you can use the Levene's test.

However, all this depends on whether the multiple measurements (the time series) are only used to account for measurment error or if they have a specific purpose (e.g. track a change over time during an experiment). For instance, if you want to model how the difference between populations changes over time you need to take the autocorrelation between time points into account. So the question is; why have you taken multiple measurements for each individual?

Basic code for the Levene's test (on means of individuals) would be:

library(car)
leveneTest(indmean ~ population, data= datafile,center="median")


For a preliminary/simple test of difference in individual variance between groups you could use the same test, but with individual variance as response. However, if the main aim is to model within-individual variance and how this differs between groups, a hierarchical random effects model is probably more suitable (nested as observations within individuals within groups).

-
Thanks for your detailed comment. You are correct that the data are normalized - in essence each distribution is just the deviation from the mean of that particular time series (so, the spread of the data). I am not interested in tracking changes over time, just whether there are statistical differences between the groups. Each time series is the output from a physical process that we have recorded. It sounds like the Levene test may have some potential, although it is not clear how this could be used to test across the two groups, as opposed to within them. –  allhands Nov 20 '13 at 18:40
The Levene's test is specifically to test for a difference between groups - I have added an example from R. –  fileunderwater Nov 20 '13 at 20:32
@allhands Also, are you only interested in the difference in variance between groups or also differences in mean value? –  fileunderwater Nov 21 '13 at 8:41
Thanks for your sample code. I'm interested in any measure that can be used to characterize differences between the plotted distributions. I'm not sure that mean will be useful in this context, as the data are standardized about their means (thus should have mean zero). –  allhands Nov 21 '13 at 9:50
@allhands No, differences in mean will naturally not be useful for standardised data (but I assume that you also have raw data). So to sum up; you are mostly interested in the within-individual variance (ie what you have plotted) and how this differs between groups(?). To look at how the variance in mean values differs between groups (i.e within-group variance between individuals) you need unstandardised data. –  fileunderwater Nov 21 '13 at 13:43

I think your confusion lies in framing your problem incorrectly. Think less in terms of your raw data and more in terms of the entities that characterize your study.

You have two populations. These populations are defined by some features. You want to know if these populations are different.

Let's consider first a simpler case where each population is defined by only one feature. The obvious choice here would be a t-test. Now let's consider the choice where each population is defined by two features. Each individual in your dataset is a point in two dimensional space: you don't have "two distributions" for each individual, the individuals are being drawn from a two-dimensional multivariate distribution. Similarly, in your current problem setup, I contend that you are looking for a test to compare the means of two 15-dimensional multivariate distributions. I believe the appropriate test here is the two-sample Hotelling's T-square test, which is the multivariate analog of the two-sample Student's t-test.

If you are unsure, consider comparing your experiment with the example described here.

-
The example is dealing with measurements of multiple variables from each sample/observation (in the example bank notes). This Q is dealing with multiple measurements of the same variable in each individual. To me, it therefore makes more sense to think about the observations in terms of variance within individuals and variance between individuals within groups. However, I agree that the framing and overall goal of the Q are a bit unclear. –  fileunderwater Nov 21 '13 at 15:22
Yeah... you've got me worried that I still don't really grock the problem/data. Maybe we should ask OP to provide a sample dataset? –  David Marx Nov 21 '13 at 17:22
As @fileunderwater points out, my question concerns repeat measurements of the same variable (that have been normalized). I have two populations, each made of $n$ individuals. I'd like to know if individuals within each population are the "same", and also if there are any significant differences between the populations (see my response to fileunderwater below for more detail). If needed, some sample data on a minimal working set could be provided. –  allhands Nov 22 '13 at 13:07