2
$\begingroup$

I would like to calculate the overlap between two distributions for which I have samples but not the PDF.

If I had the PDF of my two variables f and g, then the OVL would be $∫min(f(x),g(x))dx$. However, I do not have the PDFs, I have arrays of samples that were drawn from unknown PDFs.

So, taking my starting point to be two arrays arr1 and arr2, I have coded the following in Python:

import numpy as np
def OVL_two_random_arr(arr1, arr2, number_bins):
    # Determine the range over which the integration will occur
    min_value = np.min((arr1.min(), arr2.min()))
    max_value = np.min((arr1.max(), arr2.max()))
    # Determine the bin width
    bin_width = (max_value-min_value)/number_bins
    #For each bin, find min frequency
    lower_bound = min_value #Lower bound of the first bin is the min_value of both arrays
    min_arr = np.empty(number_bins) #Array that will collect the min frequency in each bin
    for b in range(number_bins):
        higher_bound = lower_bound + bin_width #Set the higher bound for the bin
        #Determine the share of samples in the interval
        freq_arr1 = np.ma.masked_where((arr1<lower_bound)|(arr1>=higher_bound), arr1).count()/len(arr1)
        freq_arr2 = np.ma.masked_where((arr2<lower_bound)|(arr2>=higher_bound), arr2).count()/len(arr2)
        #Conserve the lower frequency
        min_arr[b] = np.min((freq_arr1, freq_arr2))
        lower_bound = higher_bound #To move to the next range
    return min_arr.sum()

I have validated this function with some known OVL results. For example, @wolfgang proposes here some R code that calculates an OVL of 0.6099 for two normal distributions with the following properties N1(mu=1, sd=1), N2(mu=2, sd=2).

When I use my function, it yields the following results:

arr1 = np.random.lognormal(loc=1, scale=1, size = 100000)
arr2 = np.random.lognormal(loc=2, scale=2, size = 100000)
OVL_two_random_arr(arr1, arr2, 100)

==>0.65658

OVL_two_random_arr(arr1, arr2, 1000)

==>0.61107

OVL_two_random_arr(arr1, arr2, 10000)

==>0.607089

In other words, the size of the bins I use influence the results.
What should I do to avoid this? Is there a way to estimate the best number of bins (the same way we have Scott's rule and Silverman's rule for KDE)? Is there a way to avoid setting the number of bins completely?

Improve this question
$\endgroup$
9
  • $\begingroup$ Could you provide a description or mathematical definition of what you mean by "overlap" of random variables? $\endgroup$ – whuber♦ Mar 14 '17 at 18:28
  • $\begingroup$ @whuber Done, using the mathematical representation you yourself had given in another question here $\endgroup$ – MPa Mar 14 '17 at 19:05
  • $\begingroup$ Thank you. It's now evident this is not about random variables, but rather it's only about distributions. Out of curiosity, what are you hoping this calculation will tell you about your data? $\endgroup$ – whuber♦ Mar 14 '17 at 19:23
  • $\begingroup$ I'm hoping to tell whether two distributions are "similar enough". One is hard to generate, the other is easy, and I want to get a feel for whether the simplification can be a good approximation. $\endgroup$ – MPa Mar 14 '17 at 19:44
  • 2
    $\begingroup$ The OVL does not supply a meaningful number. Moreover, since you state you "want to get a feel," a probability plot will be far better than a single number. $\endgroup$ – whuber♦ Mar 15 '17 at 13:03

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.