I would like to calculate the overlap between two distributions for which I have samples but not the PDF.
If I had the PDF of my two variables f and g, then the OVL would be $∫min(f(x),g(x))dx$. However, I do not have the PDFs, I have arrays of samples that were drawn from unknown PDFs.
So, taking my starting point to be two arrays
arr2, I have coded the following in Python:
import numpy as np def OVL_two_random_arr(arr1, arr2, number_bins): # Determine the range over which the integration will occur min_value = np.min((arr1.min(), arr2.min())) max_value = np.min((arr1.max(), arr2.max())) # Determine the bin width bin_width = (max_value-min_value)/number_bins #For each bin, find min frequency lower_bound = min_value #Lower bound of the first bin is the min_value of both arrays min_arr = np.empty(number_bins) #Array that will collect the min frequency in each bin for b in range(number_bins): higher_bound = lower_bound + bin_width #Set the higher bound for the bin #Determine the share of samples in the interval freq_arr1 = np.ma.masked_where((arr1<lower_bound)|(arr1>=higher_bound), arr1).count()/len(arr1) freq_arr2 = np.ma.masked_where((arr2<lower_bound)|(arr2>=higher_bound), arr2).count()/len(arr2) #Conserve the lower frequency min_arr[b] = np.min((freq_arr1, freq_arr2)) lower_bound = higher_bound #To move to the next range return min_arr.sum()
I have validated this function with some known OVL results. For example, @wolfgang proposes here some R code that calculates an OVL of 0.6099 for two normal distributions with the following properties N1(mu=1, sd=1), N2(mu=2, sd=2).
When I use my function, it yields the following results:
arr1 = np.random.lognormal(loc=1, scale=1, size = 100000) arr2 = np.random.lognormal(loc=2, scale=2, size = 100000) OVL_two_random_arr(arr1, arr2, 100)
OVL_two_random_arr(arr1, arr2, 1000)
OVL_two_random_arr(arr1, arr2, 10000)
In other words, the size of the bins I use influence the results.
What should I do to avoid this? Is there a way to estimate the best number of bins (the same way we have Scott's rule and Silverman's rule for KDE)? Is there a way to avoid setting the number of bins completely?