How to compare two datasets in terms of density?

Question

I have two 1-D datasets (A: 300,000 points and B: 30,000 points) representing genetic events along the human genome (size: 3 billions "points"). I know that A and B are not uniformly distributed along the genome. And I know that A seems to be more clustered in hotspots than B. How can I test that A is more "densely" distributed than B?

---

Edit : I was thinking to report for each point in the dataset the minimal distance with its closest neighbor (each point has two neighbor i.e. upstream and downstream). It is then possible to compare datasets by comparing respective minimal distance distribution (the lower on average, the more "dense"). Seems ok no?

---

Edit 07 12 2016 :

Here's what I did. I used the gini index as follow. First I have to transform my list of genomic positions to something usable for a gini index analysis (a discrete distribution - each position will be assign to a "group"). In order to assign each positions to one group, I first estimate the background distribution of positions by randomly assigning X positions (where X=number of position in A), and compute the degree of each position (degree=number of position within a defined window centered on the position of interest ; for my analysis I choose 25,000). I save the maximum degree from the X positions. I do that N=100,000 times to have an estimation of the background position degree distribution. Now with my real dataset, I compute the degree for each position in A and compute an associate p-value regarding the background position degree distribution computed before ( p_i=sum(posDegree_i <= backgroundDegree)/N where i=position i in A). Then I assign adjacent positions harboring a p-value <= 0.05 to the same group and position with p-value >= 0.05 will be alone in a group). Then I have the distribution of positions by groups as:

    group      number of position
    Group_1                    10
    Group_2                     1
    Group_3                     3
    Group_4                    23
    etc ...

With this distribution I can compute a gini index indicating the relative density of the dataset A (0=no density at all ; 1=max density with only one group containing all positions i.e. one big hotspot). An issue here is that I have to fix a window size (here 25,000) that may bias the analysis..

In order to compare A and B, my idea was to subsample Nsub=10,000 times A to the size of B (30,000 points) and to compute the gini index as explained above. But I'm not sure now how to compute a final p-value to compare A and B as I will have one gini index for B and 10,000 gini indexes for A (A subsampled to B 10,000 times).

Do you think this approach is correct ?

Answer 1

One possibility resides in using appropriate sparsity indices. Such indices should encode some prior knowledge or assumptions you may have on the datasets or the information you want to reveal. The Gini index is a standard measure of inequality among different income distributions, and can be used on other data as well.

The paper Comparing Measures of Sparsity, N. Hurley & S. Rickard, 2009, first discusses six properties or axioms a sparsity measures could satisfy, and probes some known measures against them. The abstract is:

Sparsity of representations of signals has been shown to be a key concept of fundamental importance in fields such as blind source separation, compression, sampling and signal analysis. The aim of this paper is to compare several commonly-used sparsity measures based on intuitive attributes. Intuitively, a sparse representation is one in which a small number of coefficients contain a large proportion of the energy. In this paper, six properties are discussed: (Robin Hood, Scaling, Rising Tide, Cloning, Bill Gates, and Babies), each of which a sparsity measure should have. The main contributions of this paper are the proofs and the associated summary table which classify commonly-used sparsity measures based on whether or not they satisfy these six propositions. Only two of these measures satisfy all six: the $pq$-mean with $p<1$, $q>1$ and the Gini Index.

Your data might requires some preprocessing or transformation before sparsity indices could be applied successfully.