# Compare in-pairs differences to between-pairs differences

I have some numerical data which comes in pairs. I need to test the hypothesis that values inside pairs are more similar to each other, than to other values in the sample. Is there a test for this?..

What I came up with is calculating differences in each pair and then taking variance of this sample. Then move second values in each pair, so that all pairs are disrupted and the values are paired to a wrong value. Then take the variance of differences. And do this until all such shifts without returning to the original pairs are performed. Then compare this "sample" of variances to the original "true" variance using t-test/wilcoxon test. Is this a statistically correct procedure?

EDIT:

Example data:

0.225610  0.512543
0.301496  0.414367
0.451774  0.406694
0.102956  0.220093
0.365923  0.463573
0.328177  0.318119
0.156525  0.281603
0.569386  0.259615
0.409145  0.380789
0.346699  0.184932
0.148997  0.256125
0.196723  0.241247
0.164317  0.255147
0.323110  0.268701
0.470532  0.426732
0.491121  0.674537
0.417971  0.358050
0.562317  0.481768
0.355106  0.558749
0.574543  0.158430
0.234307  0.482079
0.578360  0.092736


My hypothesis is that on average values in each pair are more similar to each other, than to values in other pairs. What I am doing now, is shifting all values in the second column by one, calculate differences and then take the variance of these differences. Then I repeat the procedure, but shift by 2, etc. Then compare this set of artificial variances to the variance of original data.

(Order in pairs doesn't matter)

• Do you mean that the difference in the original pairs is lower than if the rows are randomly shuffled (and destroying the original pairs)? Apr 14, 2016 at 16:54
• Yes, exactly, that's what I mean. Apr 14, 2016 at 16:55
• This sounds like an optimization problem to me. Let me think about how to approach it and formalize it. Apr 14, 2016 at 16:58

It sounds like you're maybe describing a version of a permutation test

If you have a null/alternative pair like this:

$H_0: Var(X_i-Y_i)=Var(X_i-Y_j)$ (for $i\neq j$)

$H_1: Var(X_i-Y_i)<Var(X_i-Y_j)$

then it looks like you could perhaps do something similar to what you have in mind.

Note that (assuming that cross pair observations are independent), this is equivalent to testing whether the correlation between paired $X$ and $Y$ is positive (since $Var(X_i-Y_j) = Var(X_i)+Var(Y_j)$ while $Var(X_i-Y_i) = Var(X_i)+Var(Y_i)-2Cov(X_i,Y_i)$, so if the covariance is positive the variance of the difference will be reduced.)

As such, one possibility is simply to test correlation (which again, could be done via a permutation test if desired).

• Thank you for your suggestion! Using correlation instead of variance of differences is more obvious (but gives the same result, actually). Apr 15, 2016 at 10:20

Yes, absolutely! Classically one might use the t-test but there are far more powerful tools developed fairly recently. Using Bayesian approach, one can reason not simply about failure of rejection/rejection at some p-value, but reason about the whole spectrum. At some confidence level, one might even accept the null hypothesis! The power lies in natural handling of uncertainty which is lacking in frequentist methods. I strongly suggest using the BEST method by Kruschke John [PDF]. The paper in a fairly extensive manner explains the downsides of the frequentist approach with the t-test and then discusses the proposed Bayesian approach.

The implementation in R, I believe, can be found on authors website [WWW]. If you are more comfortable with Python, I suggest to look at PyMC which implements this method in the examples section [HERE]. You can plug in your data and start playing with it right away.

I am fairly new to Probabilistic Programming but reading numerous tutorials and playing with data, can tell that Bayesian approach to medium and especially small sample sized datasets is very powerful. This is where the Bayesian approach to data analysis excels.

If the number of data pairs is relatively small, say in order of hundreds, I would try to compute the average distances for randomly shuffled columns and compare it to your reference. Here I assume that your dataset comes originally with some ordering and you are wondering whether its current configuration is optimal in a sense you described it.

If the number of pairs is low, you can try to compute the measure for all possible pairings. If the number of pairs is large, you can draw some number of random pairings. In the latter case, you could try to visualize the distribution of the measure and see whether there is some structure.

• Thank you, xeon! I'll consider using something other than classical test! I actually have very small datasets. I just don't see how this method is applicable to comparing a sample to one value?.. Or am I missing something? Apr 14, 2016 at 16:28
• @Phlya Re-reading your question more carefully reveals that I didn't understand your question well enough. I thought this was about classical two group hypothesis. I think that in the first place you need to make clear what exactly you are looking for in the data (the question) and model it appropriately. The beauty of Probabilistic Programming is that modeling and inference tasks are nicely decoupled. Apr 14, 2016 at 16:34
• @Phlya Do you mind sharing the data and extending little bit more your question about your expectations and analysis you would like to do? Apr 14, 2016 at 16:35
• I've edited my question. Apr 14, 2016 at 16:40