I have a set of ten samples, comprised of five pairs of twins, and I have calculated the pairwise distances between all ten samples. The question I would like to answer with this data is:
Are twins more similar to each other than to a random sample
or, more formally
Are the distances between paired samples significantly lower than the distance between unpaired samples
On the face of it, it seems like an easy comparison: all paired distances go in one bin, all unpaired go in the other, and compare. However, I recognize that this breaks all kinds of rules regarding independence. A quick simulation in R suggests that including all pairwise distances significantly under-estimates the significance of the p-values for the comparison. Unfortunately, I have no idea how to solve this problem.
I can think of a number of options, such as: only including one non-twin comparison for each twin pair, averaging all non-twin distances for each sample, or comparing each point to a centroid, but none of these seem entirely reasonable. Is there a standard way for doing such a comparison? Or are pairwise distances simply not the right way to answer our hypothesis?
EDIT: The overall study design is a bit complicated, which is why I didn't post it here originally. However, if working with distances is not possible, then it will be necessary to consider exactly what data I'm working with. I'm including a quick description of the study design below:
We're working on something very similar to a RNA-seq study, where RNA is collected from each of our samples, amplified, and then sequenced (note that only a small portion of the total RNA pool is sequenced). After sequencing, you have a random number of molecules from each sample which we bin according to which gene the RNA molecule was produced from. The output is a sample-by-gene count matrix, which we want to use as an estimate for the relative likelihood that each gene will be transcribed into RNA. Generally, these counts are modeled by the combination of two distributions: a Gamma distribution for the biological noise, and a Poisson distribution for the number of molecules actually sequenced. The resulting distribution is usually modeled as a negative binomial. Whether these assumptions hold for our own data is debatable, but for now I'm going to assume they do.
Making the above assumptions, there are some well-established methods for group comparisons of individual genes. However, quantifying the overall similarity of samples is apparently not common. The major problem with any standard measures of similarity is that variance increases in samples with fewer total molecules, and the samples with the highest total number of molecules are always most similar. So currently, to estimate the differences between samples, we are subsampling down to the smallest total read count and calculating distances on the resulting count matrix. The subsampling is then repeated N times, and an average distance matrix is calculated.
There's probably better ways to quantify the overall similarity, but they need to take these distributions into account.