Ranking distributional data by similarity

Question

The question in short: What methods can be used to quantify distributional relationships between data when the distribution is unknown?

Now the longer story: I have a list of distributions and would like to rank them based on their similarity to a given base-line distribution. Correlation jumps into my mind in such a case and the Spearman correlation coefficient in particular given that it does not make any distributional assumptions. However, I would actually need to create the coefficient based on binned data (as this is done for histograms or densities) rather than the raw data, and I don't know if this is actually a valid step or if I am just manufacturing data.

In other words, if I have a 10000 point data set for each distribution, I would first create a binned distribution for each were each bin is of equal width and contains the frequencies of how many points each bin has. Just the way this is done for density plots or histograms. Each bin is on a discrete scale. The data is actually computer screen coordinate data and values are between 1 and 1024. Each pixel position could represent a bin (but larger bins are possible e.g. every 5 pixel being one bin). I would then compare the sequence of bins with each other rather than the raw data. The data set would look like this.

bins: 1 2 3 4 .... 1024<br>
dist#base:1 2 2 3 ..... 3<br>
dist#1: 1 4 5 5 3<br>
dist#2: 2 2 3 5 6<br>
...<br>
dist#1000: 1 2 4 6 6<br>

Does this make sense? Are there better ways of doing that? Are there better statistical methods? The goal of all this is to first) test how close are distributions from measure A to measure B and second) if I can predict one, if the other is missing.

Answer 1

I think relative distribution methods are a good candidate for the question you pose.

Since you're comparing data based on binning, this is very similar to the method of constructing a probability-probability plot. Taking it a step further, you can can actually construct a relative CDF/PDF for two distributions based on empirical data. From there, you can apply graphical and statistical techniques to explore the relative differences and perform inference on the relative distribution.

Handcock and Morris have an interesting book devoted to this topic and there's the reldist package in R available for applying these methods. The following might be worth skimming to see if this is of any interest:

• Relative Distribution Methods
• Applying Relative Distribution Methods in R

Answer 2

you could compute a Kolmogorov-Smirnov statistic based on your binned data. This would work by first computing an empirical CDF based on your bins (just a cumulative sum with rescaling), then compute the $\infty$-norm of the differences.

I don't know R well enough to give you code in R, but can quote the Matlab very simply:

%let base be the 1 x 1024 vector of binned observed data
%let dists be the 1000 x 1024 matrix of binned distributions to be checked
emp_base = cumsum(base,2);emp_base = emp_base ./ emp_base(end);  %cumsum, then normalize
emp_dists = cumsum(dists,2);
emp_dists = bsxfun(@rdivide,emp_dists,emp_dists(:,end));   %normalize for the top sum.
emp_diff = bsxfun(@minus,emp_base,emp_dists);   %subtract the cdfs; R does this transparently, IIRC
KS_stat = max(abs(emp_diff),[],2);   %take the maximum absolute difference along dim 2
%KS_stat is now a 1000 x 1 vector of the KS statistic. you can convert to a p-value as well.
%but you might as well just rank them.
[dum,sort_idx] = sort(KS_stat);  %Matlab does not have a builtin ranking; PITA!
dist_ranks = nan(numel(sort_idx),1);
dist_ranks(sort_idx) = (1:numel(sort_idx))';
%dist_ranks are now the ranks of each of the distributions (ignoring ties!)
%if you want the best match, it is indexed by sort_idx(1);

the bsxfun nonsense here is Matlab's way of doing proper vector recycling, which R (and numpy, IIRC) does transparently.

Answer 3

Edit: I misunderstood the question at first. Your observations are actually paired. Sample1 Bin1 to Baseline Bin1 etc.

What you could do is take the difference between sample and baseline for each bin, then use the Wilcoxon signed rank statistic on the differences.

http://en.wikipedia.org/wiki/Wilcoxon_signed-rank_test

If

• D is your sequence of differences, then
• R is the ranks of |D|, and
• psi = 0 if D < 0, 1 if D > 0.
• W = Sum(psi*R)

In R

wilcox.test(sample1-baseline)$statistic

Answer 4

You have two good choice of metrics here, one is the Kullback-Leibler Divergence between your distributions and your baseline distribution. The KL divergence essentially gives a metric of the "distance" between two probability distributions and has nice information theoretic interpretations.

The second one is the Kolmogrov-Smirnov test, as pointed out by @shabbychef.

For related posts on these distribution metrics, see this and this.

Since you mentioned that you want to know

test how close are distributions from measure A to measure B and second) if I can predict one, if the other is missing.

I would recommend the KL-divergence, as your second requirement can be interpreted as asking for the mutual-information between A and B.