# How to compare two datasets in terms of density?

I have two 1-D datasets (A: 300,000 points and B: 30,000 points) representing genetic events along the human genome (size: 3 billions "points"). I know that A and B are not uniformly distributed along the genome. And I know that A seems to be more clustered in hotspots than B. How can I test that A is more "densely" distributed than B?

Edit : I was thinking to report for each point in the dataset the minimal distance with its closest neighbor (each point has two neighbor i.e. upstream and downstream). It is then possible to compare datasets by comparing respective minimal distance distribution (the lower on average, the more "dense"). Seems ok no?

Edit 07 12 2016 :

Here's what I did. I used the gini index as follow. First I have to transform my list of genomic positions to something usable for a gini index analysis (a discrete distribution - each position will be assign to a "group"). In order to assign each positions to one group, I first estimate the background distribution of positions by randomly assigning X positions (where X=number of position in A), and compute the degree of each position (degree=number of position within a defined window centered on the position of interest ; for my analysis I choose 25,000). I save the maximum degree from the X positions. I do that N=100,000 times to have an estimation of the background position degree distribution. Now with my real dataset, I compute the degree for each position in A and compute an associate p-value regarding the background position degree distribution computed before ( p_i=sum(posDegree_i <= backgroundDegree)/N where i=position i in A). Then I assign adjacent positions harboring a p-value <= 0.05 to the same group and position with p-value >= 0.05 will be alone in a group). Then I have the distribution of positions by groups as:

    group      number of position
Group_1                    10
Group_2                     1
Group_3                     3
Group_4                    23
etc..


With this distribution I can compute a gini index indicating the relative density of the dataset A (0=no density at all ; 1=max density with only one group containing all positions i.e. one big hotspot). An issue here is that I have to fix a window size (here 25,000) that may bias the analysis..

In order to compare A and B, my idea was to subsample Nsub=10,000 times A to the size of B (30,000 points) and to compute the gini index as explained above. But I'm not sure now how to compute a final p-value to compare A and B as I will have one gini index for B and 10,000 gini indexes for A (A subsampled to B 10,000 times).

Do you think this approach is correct ?

Thanks

• What exactly are A and B? Counts ? – utobi Dec 2 '16 at 13:22
• A and B are just genomic position in the human genome – Nicolas Rosewick Dec 2 '16 at 13:39
• Ok, just quickly three possible approaches: test for the equality of variances, permutation test for the the sum of absolute differences (SAD, stackoverflow.com/questions/5855066/…) or permutation test for the Gini concentration index. If you are curious, I can give you a full answer with R code tonight. – utobi Dec 2 '16 at 14:15
• Thanks. For the R code I'm very curious. I guess I should first transform the dataset (a list of number indicating a genomic position) in something interpretable by a gini index analysis (something like a distribution of the points in different clusters ? ) – Nicolas Rosewick Dec 4 '16 at 16:34
• Yes you can see that as points on a line (the line is finite - around 3 billion long ) – Nicolas Rosewick Dec 4 '16 at 17:07

Sparsity of representations of signals has been shown to be a key concept of fundamental importance in fields such as blind source separation, compression, sampling and signal analysis. The aim of this paper is to compare several commonly-used sparsity measures based on intuitive attributes. Intuitively, a sparse representation is one in which a small number of coefficients contain a large proportion of the energy. In this paper, six properties are discussed: (Robin Hood, Scaling, Rising Tide, Cloning, Bill Gates, and Babies), each of which a sparsity measure should have. The main contributions of this paper are the proofs and the associated summary table which classify commonly-used sparsity measures based on whether or not they satisfy these six propositions. Only two of these measures satisfy all six: the $pq$-mean with $p<1$, $q>1$ and the Gini Index.