How to assess whether weighted samples come from the same distribution? Suppose I have two samples, and I'd like to compare them to see if they come from the same underlying distribution. Normally, I would probably do this using a KS test. However, suppose also that I have some 'weight' for each observation in each of the two samples indicating how 'sure' I am that the sampled value is 'accurate'. In other words, some observations in the samples are more representative of the sampled distribution than others (note that these weights will differ between samples).
With this in mind, my question is: Are there any tests (like a KS test) that I can use to test if these two samples come from the same underlying distribution, taking into account the weights associated with each sample?
Hopefully I'm not being too vague here... but any help and/or comments are greatly appreciated.
 A: Like Michelle I'm curious about how this situation comes about.  But putting that aside, what about putting a bootstrap around a KS test, with the bootstrap resampling based on your weights?  So those points you are more confident about are more likely to get resampled.
A: I just had a simple thought that might work.  I assume the weights are known and fixed. Modify the samples to make the distributions look like the weighted distribution. The observations with the smallest weight enter once.  Then an observation with twice that weight gets entered twice etc.  After creating the modified sample apply a standard two sample goodness of fit test. Because the distributions are continuous you could modify this to make the duplicates differ from the original by adding a small random component to each. Now the only problem I see is that every weight have to be an integer multiple of the point(s) with the lowest weights.  To remedy this you could make the lowest weight observations repeat say 5 times.  Then the others could be fractional multiples of the smallest weighted observation.
