What would be the correct (most statistically sound) way to tackle the following problem:

Given are two samples (or empirical distributions) from two unknown distributions $X$ and $Y$, with $N_x$ and $N_y$ sample points, respectively.

I would like to compute a sample (or empirical distribution) of the difference (according to any metric, e.g. MSE, Delta E, ...) between the two distributions based on the corresponding samples (or empirical distributions). It is important to note that for practical reasons the difference (based on the used metric) can only be computed between two individuals sample points.

I currently see only three practical options (illustrated in the embedded image):

  1. $N_x : N_y$: compute the difference between all unique combinations of the sample points taken from the two distributions.
  2. $1 : 1$: compute (without replacement) only pairwise differences.
  3. $random$: compute randomly (with replacement) a certain number of sample points of the difference.

enter image description here

Option 1 would certainly squeeze as much information from the given samples as possible. Option 2 would be much more convenient for “online” computations. Option 3 has the possibility to randomly sample the same pair.

I am interested in the most efficient and statistically sound way to compute this sample (or empirical distribution) of differences, as I would like to subsequently use it in an one-sample equivalence test (TOST) to test for $\mu = 0$.

Unfortunately, I am not a statistician and therefore, not sure if some option introduces unwanted bias or side effects. If you consider the extreme case having $N_x = 1$ and $N_y = 100$ then the first distribution $X$ is obviously not well represented.

I am not sure what the statistical term for the thing I am trying to accomplish is and therefore I am not sure where to look for the appropriate theory. It seems a little bit like bootstrapping, but then again, not quite.

Any feedback, suggestions and hints in the right direction are appreciated.

EDIT: (regarding comment by whuber)

I am basically interested in the difference (according to some metric e.g., Delta E) between the means of the two underlying distributions. In particular, I would like to test for equivalence of the two means. Normally, one would probably use a two-sample TOST in this case. But since my sample points are RGB vectors and the Delta E metric is a non-trivial difference between those vectors, I am not sure if it is possible (or how) to adapt such a test. Therefore, I am now trying to compute a sample of the difference between the two arbitrary distributions (i.e., empirical samples) and use this in a one-sample TOST to test for $\mu = 0$.

  • $\begingroup$ What does your data represent? $\endgroup$ – Tim Jan 8 '18 at 15:16
  • $\begingroup$ The individual sample-points are radiance values (RGB pixels) computed by a renderer using Monte Carlo integration. The underlying distributions are in general arbitrary. $\endgroup$ – Christian F. Jan 8 '18 at 15:33
  • $\begingroup$ Re: "It is important to note that for practical reasons the difference (based on the used metric) can only be computed between two individuals sample points." Then please explain how that difference would be computed for the two underlying distributions, which don't involve sample points at all. That will determine the appropriate way to resample. $\endgroup$ – whuber Jan 8 '18 at 17:56
  • $\begingroup$ @whuber Please see my EDIT above. $\endgroup$ – Christian F. Jan 9 '18 at 14:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.