Can I use a permutation test to determine whether overlap of 2 distributions is significant? I am evaluating whether two distributions are different from each other.  For example, histogram A (B) has most of its binned values between 10-15 (15-20).  So, histogram B has a right-ward shift in its distribution versus histogram A.  However, is the B distribution significantly offset from the A distribution?
First, I have normalized the original histograms to create two PDFs with both sums = 0:
PDF1 = (0, 0, 0, 0.2, 0.25, 0.3, 0.15, 0.1, 0, 0)
PDF2 = (0, 0, 0, 0, 0.3, 0.15, 0.2, 0.2, 0.10, 0.05)
Then, I created an overlap parameter (range 0-1) for the 2 original sample PDFs, calculated by summing the minimums of each bin: 0+0+0+0+0.25+0.15+0.15+0.1+0+0 = 0.6.  So, the 2 sample PDFs overlap each other by 0.6.
As the sample size is not large, and I would like to have robust statistics for determining whether the 2 distributions are significantly different (offset), I computed permutations (1,000 for each sample) of the two PDFs.  Then I computed the 1,000 overlap values from each of the resampled PDFs.  Bootstrapping was not appropriate as it resulted in some resampled PDFs that contained all 0s or a sum > 1, all of which is not possible.
H_0: The distributions are the same.
H_A: The distributions are different.
Questions:


*

*Is the above procedure fair and appropriate?

*How would I go about obtaining a p-value and confidence interval, and conclude the original sample overlap value is significant or not?  I am only interested in a 1-tail (left) test, as the lack of overlap is important versus having too much overlap.


Any suggestions for improvement or alternative methods would be appreciated.  I am using a combination of Python and R.
 A: IIUC, you can use a permutation test relatively directly on the problem.
For $A$ and $B$, define whatever you mean by overlap. For your case, this could be, e.g.,


*

*the number of elements in $A$ higher than the smallest element in $B$; or, as a generalization - 

*the number of elements in the $1 - \alpha$ lower quantile of $A$ higher than elements in $B$'s $1 - \alpha$ higher quantile (as a concrete example: the number of the lowest 99% elements in $A$, higher than the highest 99% elements in $B$)
Call this $o = \text{Ov}(A, B)$. 
Now repeat $b$ iterations. In each iteration $i$, permute the elements, and choose $A'_i$ and $B'_i$ as groups having respectively $|A|$ and $|B|$ elements. Find $o_i = \text{Ov}(A_i, B_i)$ for the iteration. Finally, find the fractional rank of $o$ in $\{ o_1, \ldots, o_b \}$, and that is the p-value.
A: Almost any reasonable statistic you can compute on two samples could be used. 
One could base it off the overlap in a KDE for example. 
For data with no ties, I think you can get directly at a measure of overlap independent of the choice of bins, by making a function that computes for any cutoff point $k$ 
$$\min_k(\,p(A<k)+p(B>k),\,p(A>k)+p(B<k)\,)$$
where $p$ is "the proportion of times its argument is true" (in R you do that by replacing p by mean). I think the minimizing over all possible $k$ could at least be accomplished via trying one value of $k$ between each pair of adjacent observations in a combined sample.
I believe this is equivalent to a test based purely on ranks (as far as I see the statistic only depends on the ordering of the combined samples and so would not be affected by any monotonic increasing transformation), which suggests that it may be possible to circumvent the necessity of doing all the permutations -- in small samples the distribution of the statistic across all rank permutations should be calculable and in large samples there should be an asymptotic approximation.
Either way this can then be the statistic in a permutation test; I see nothing to suggest it would be unsuitable.
(Actually I'd be somewhat tempted to replace this with a version of the Wilcoxon-Mann-Whitney statistic that's kind of similar, but I don't know if it meets your needs)
[If you can't get original data and must use histograms then how it works depends on whether the histogram bins are defined independently of the data and you'll have a number of other small issues to deal with, but I don't think any of those issues lead to problems you can't resolve.]
Let's look at a teeny example. Take two samples
  A         1.59        2.13 3.54 
  B   1.32        1.68             7.20 9.55

 This corresponds to the sample order B A B A A B B
 We can then look at each of the cutpoints

         B A B A A B B
        | | | | | | | |
    A<  0 0 1 1 2 3 3 3
    B>  4 3 3 2 2 2 1 0
    A>  3 3 2 2 1 0 0 0
    B<  0 1 1 2 2 2 3 4

  A<+B> 4 3 4 3 4 5 4 3 
  A>+B< 3 4 3 4 3 2 3 4 

  Min   3 3 3 3 3 2 3 3
                  ^
     Statistic is 2

Noting that the sum of the two things we're taking the min over is always $N=n_A+n_B$, we can just compute one of the two and take min(min,N-max) within that one. So it's sufficient to either consider  "A< + B>" or "A> + B<" and just find both the min and max of that. E.g. if we look at the "A<+B>" row,  4 3 4 3 4 5 4 3 it has min 3 and max 5, so min(3,7-5)=2.
Now we have a statistic we can compute it for all orderings. The smallest achievable significance level in this tiny case is 2/35 (0.057), which is complete non-overlap of samples (statistic 0; AAABBBB and BBBBAAA), so we wouldn't reject H0 for our observed sample on that statistic.
If you're generating permutations the ties you have won't be an issue; everything still works; you just count how many samples are at least as extreme -- the same or smaller overlap -- as yours. You would include the observed sample in the total set (which just increases numerator and denominator by 1)

With the confidence interval I'm not sure if this is what you actually seek, but a confidence interval would be for a parameter describing a particular kind of alternative that can be indexed by that parameter. 
For example, you could get a confidence interval for a location shift easily enough. You identify the interval for the shift parameter in which there's no rejection by shifting the second sample left and right to the limit of where you switch from non-rejection to rejection, so you get the full length of the interval that just gives non-rejection at level no more than some $\alpha$; this should then give you a $1-\alpha$ interval for the shift parameter. 
If you're seeking an interval for the overlap itself I don't know how this can be done within this suggested permutation test because there's no direct way to manipulate the sample overlap directly like that. Under some parametric assumption you may be able to get somewhere perhaps.
