In summary: I'm trying to compare the distribution of two samples containing measurements with continuous likelihood distributions rather than precise measurements. In my head it makes sense to do this with a modified KS test, by constructing cumulative distribution functions (CDFs) from the underlying likelihood distributions of the individual samples' elements. Is this OK? Is there some well established method of doing this? Is there another statistical test I should be looking at?

In full: I've got two different samples consisting of several dozen galaxies. Each galaxy in the samples has an estimation of its distance. The two samples were selected in slightly different ways, so it's not clear if the different selection methods lead to differently biased samples. What I would like to do, is look at the distance distribution for each and determine whether I can reject the null hypothesis that both sets are drawn from the same underlying population distance distribution. What normally springs to mind is to construct two empirical cumulative distribution functions and compare them with a two sample Kolmogorov-Smirnov (KS) test. There is a catch though: even though each galaxy has some kind of a distance estimation, it's not a single precise value. Because of the way these distances were estimated, we instead end up with a distance likelihood distribution for each galaxy. These distributions are definitely not Gaussian, in fact sometimes they've even got more than one mode (eg. for one galaxy we might have a most likely distance value of 100 kpc which tails off at higher and lower values, but then spikes up to another peak at 25 kpc).

With the normal KS test I would construct an empirical cumulative distribution function for the distances in each of my samples and then find the biggest difference between them, D. In my situation where each galaxy has a distance likelihood distribution rather than a precise value, I can still construct the sample's cumulative distribution function (essentially by co-adding each galaxy's likelihood distribution to create the sample's overall distance density distribution, and then turning this into a cumulative distribution function.) I can then compare the two samples' CDFs and find the biggest difference, or the D statistic. It's not clear to me whether this still counts as a KS test.  Have I committed any big sins yet? If this is still OK, then how do I evaluate the null hypothesis with my D value? Can I do it the normal way which depends on the number of entries in each sample, (Neffective=(N1*N2)/(N1+N2)) or does that change when I construct my continuous CDF?

  • $\begingroup$ It seems that your sampling is insufficiently defined for the results of a test to be in any way definitive, or even helpful. I would suggest that you simply plot the likelihood functions and decide whether they are distinctly similar or different. (Perhaps you should conduct a simulation study to get a feel for how the likelihood functions vary before making inferences about the real data.) $\endgroup$ – Michael Lew Aug 29 '16 at 21:08
  • $\begingroup$ @Michael Thanks for comment. I'm curious what you mean by insufficient sampling. Do you mean the sampling of the distance? If so, I'm not too worried. The resolution of the distance likelihood function is decent. When I said above that a likelihood function might "spike" at 25 kpc I didn't mean to imply that it isn't smoothly varying (it is), nor that this is typical (most are better behaved). Although I didn't mention it, the distance models (and the associated likelihoods) have been well characterized elsewhere, and the asymmetry in the likelihood distributions is a well known effect. $\endgroup$ – Drew Aug 29 '16 at 22:04
  • $\begingroup$ I was referring to your statement: " The two samples were selected in slightly different ways, so it's not clear if the different selection methods lead to differently biased samples." If you don't know whether there is differential bias in the data then you should not be too quick to take those data at face value. A relatively subjective inspection of the likelihood functions may be safer than a more formal test. $\endgroup$ – Michael Lew Aug 30 '16 at 1:29

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.