In summary: I'm trying to compare the distribution of two samples containing measurements with continuous likelihood distributions rather than precise measurements. In my head it makes sense to do this with a modified KS test, by constructing cumulative distribution functions (CDFs) from the underlying likelihood distributions of the individual samples' elements. Is this OK? Is there some well established method of doing this? Is there another statistical test I should be looking at?
In full: I've got two different samples consisting of several dozen galaxies. Each galaxy in the samples has an estimation of its distance. The two samples were selected in slightly different ways, so it's not clear if the different selection methods lead to differently biased samples. What I would like to do, is look at the distance distribution for each and determine whether I can reject the null hypothesis that both sets are drawn from the same underlying population distance distribution. What normally springs to mind is to construct two empirical cumulative distribution functions and compare them with a two sample Kolmogorov-Smirnov (KS) test. There is a catch though: even though each galaxy has some kind of a distance estimation, it's not a single precise value. Because of the way these distances were estimated, we instead end up with a distance likelihood distribution for each galaxy. These distributions are definitely not Gaussian, in fact sometimes they've even got more than one mode (eg. for one galaxy we might have a most likely distance value of 100 kpc which tails off at higher and lower values, but then spikes up to another peak at 25 kpc).
With the normal KS test I would construct an empirical cumulative distribution function for the distances in each of my samples and then find the biggest difference between them, D. In my situation where each galaxy has a distance likelihood distribution rather than a precise value, I can still construct the sample's cumulative distribution function (essentially by co-adding each galaxy's likelihood distribution to create the sample's overall distance density distribution, and then turning this into a cumulative distribution function.) I can then compare the two samples' CDFs and find the biggest difference, or the D statistic. It's not clear to me whether this still counts as a KS test. Have I committed any big sins yet? If this is still OK, then how do I evaluate the null hypothesis with my D value? Can I do it the normal way which depends on the number of entries in each sample, (Neffective=(N1*N2)/(N1+N2)) or does that change when I construct my continuous CDF?