Bayesian inference of Pr(X > Y) where X and Y each have an approximate posterior distribution

Question

I am developing a Bayesian system in which I would like to quantify the evidence for or against the conclusion that one data-generating process (X, for which we observe X = x) will produce a more extreme result than another process (Y, for which we observe Y = y).

For my purposes, by "more extreme" I mean that I am interested in the quantity Pr(X > Y). Also, X and Y are defined over the positive integers, so "extreme" literally means "more positive".

Suppose that I have empirical posterior distributions for both processes X and Y, that I acquired through an MCMC procedure. Now, my understanding is that since I can sample directly from the posteriors I could simply draw pairs from X and Y an arbitrary number of times, and report the proportion of instances in which x > y. Is this correct?

As you read this question, you may infer that I am generally interested in a procedure for comparing X and Y that does not rely on a summary statistic of the two distributions (e.g., Pr(E(X) > E(Y) ). Why would I describe my posterior with a summary when I have the entire posterior distributions to work with? I am unaware of a general method for comparing two distributions that does not degenerate into a simple comparison of means (or other similar summary statistic).

Edit: a bit of context, below:

Essentially, I have a computer simulation that is designed to accept a bunch of parameters values and then simulate the number of products sold over a 3-month period of time. The parameters include things like time of year (spring, summer, winter, fall), number of stores (numeric), average income of the area (numeric), etc.

What I did is set up the simulation with two different sets of parameterizations (e.g., simulation "X" took place in wintertime in a low-income neighborhood whereas simulation "Y" took place in summertime in a high-income neighborhood), and then allowed the program to generate two sets of simulated numbers of products sold. The simulations under both conditions were replicated many times so I have a distribution of results for both X and Y.

My goal is to demonstrate that X > Y. I understand that looking at means/modes would simplify my analysis, but I don't want to make my result too sensitive to the shape of the distributions (e.g., could be polymodal).

Since the two distributions were generated independently but by programs that only differ by a few parameter values, I suspect that X and Y will vary in similar ways but can be considered to be independent from one another.

Answer 1

Yes you "could simply draw pairs from $X$ and $Y$ an arbitrary number of times, and report the proportion of instances in which $x > y$." What you wish to estimate is not a random variable; hence, it is constructed from a point estimator of a parameter of a random distribution. In this case the random distribution may either be the difference statistic $D = X - Y$ or the ratio statistic $R = \frac{X}{Y}$ and the parameters would be $P(D>0)$ and $P(R>1)$, respectively. Clearly both of these are equivalent to $P(X>Y)$. Looking at the posterior distribution of $D$ and $R$ will provide additional information into disparities of the posterior densities of $X$ and $Y$, but perhaps a simple overlay of the posterior densities would suffice.

It is important to stress that to draw a sample from the posterior density of $D$ or $R$, one merely draws a random pair from the posterior density of $(X,Y)$ and computes $D=X-Y$ or $R=X/Y$.

Answer 2

What is not clear here is what you mean by "extreme." One consideration beyond looking at a measure of centrality, however disguised, would be to examine the variances because what you are implying is comparing the volatility of X and Y and not especially their centrality. Even more brutally stated, comparing centrality doesn't work. For example, two Beta distributions, one $\cup$-shaped and the other $\cap$-shaped can have $X-Y=0$ and yet the $\cup$-shaped distribution is much more extreme. The other concept that needs clarification is that in the context you are implying, you are more concerned with the right sided tail of the distributions than than on the left and again, measures of things like skewness or upper quartile values speak more to what you seem to be asking than what you have proposed. So, back to basics; please clarify what you are asking and let us know what that is.