I would like to calculate the overlap between two distributions for which I have samples but not the PDF.
If I had the PDF of my two variables f and g, then the OVL would be $∫min(f(x),g(x))dx$. However, I do not have the PDFs, I have arrays of samples that were drawn from unknown PDFs.
So, taking my starting point to be two arrays arr1
and arr2
, I have coded the following in Python:
import numpy as np
def OVL_two_random_arr(arr1, arr2, number_bins):
# Determine the range over which the integration will occur
min_value = np.min((arr1.min(), arr2.min()))
max_value = np.min((arr1.max(), arr2.max()))
# Determine the bin width
bin_width = (max_value-min_value)/number_bins
#For each bin, find min frequency
lower_bound = min_value #Lower bound of the first bin is the min_value of both arrays
min_arr = np.empty(number_bins) #Array that will collect the min frequency in each bin
for b in range(number_bins):
higher_bound = lower_bound + bin_width #Set the higher bound for the bin
#Determine the share of samples in the interval
freq_arr1 = np.ma.masked_where((arr1<lower_bound)|(arr1>=higher_bound), arr1).count()/len(arr1)
freq_arr2 = np.ma.masked_where((arr2<lower_bound)|(arr2>=higher_bound), arr2).count()/len(arr2)
#Conserve the lower frequency
min_arr[b] = np.min((freq_arr1, freq_arr2))
lower_bound = higher_bound #To move to the next range
return min_arr.sum()
I have validated this function with some known OVL results. For example, @wolfgang proposes here some R code that calculates an OVL of 0.6099 for two normal distributions with the following properties N1(mu=1, sd=1), N2(mu=2, sd=2).
When I use my function, it yields the following results:
arr1 = np.random.lognormal(loc=1, scale=1, size = 100000)
arr2 = np.random.lognormal(loc=2, scale=2, size = 100000)
OVL_two_random_arr(arr1, arr2, 100)
==>0.65658
OVL_two_random_arr(arr1, arr2, 1000)
==>0.61107
OVL_two_random_arr(arr1, arr2, 10000)
==>0.607089
In other words, the size of the bins I use influence the results.
What should I do to avoid this? Is there a way to estimate the best number of bins (the same way we have Scott's rule and Silverman's rule for KDE)? Is there a way to avoid setting the number of bins completely?