I have two continuous random variables, $X$ and $Y$, whose distributions are unknown. I can draw samples from them and in particular, I am interested in estimating $P(X<Y)$ based on those samples. Let's assume that each sample has $n$ elements: $x_1, x_2, \ldots, x_n$ are drawn from $X$ and $y_1, y_2, \ldots, y_n$ from $Y$.

Can I use $$T=\frac{1}{n^2}\sum_{i=1}^{n}\sum_{j=1}^{n}I(x_i < y_j)$$ as the estimate of $P(X<Y)$ if I assume $X$ and $Y$ are independent and is such an estimator asymptotically normally distributed? If I knew

Also, if I assume that $X$ and $Y$ are dependent, can I use $$S=\frac{1}{n}\sum_{i=1}^{n}I(x_i < y_i)$$ as the estimator of said probability and is it asymptotically normally distributed?

I'm asking about asymptotic normality because I would like to determine lower and upper bounds of $P(X<Y)$ and think that confidence intervals can act as such.

P.S. $I(E)=1$ if $E$ is true, and $0$ otherwise.

  • 3
    $\begingroup$ You can apply the Central Limit Theorem directly to $S$ because it's the sum of bounded iid variables. (Indeed, it is proportional to a Binomial$(n, \Pr(X\lt Y))$ distribution.) It takes a little bit more work to apply it to $T$ directly because the variables, although identically distributed and bounded, are correlated. However, you can compute their variance-covariance matrix explicitly, which gives one way to draw the conclusion (as well as figure out the bounds). Of greatest interest, perhaps, is the question whether $T$ is any better an estimator than $S$. $\endgroup$
    – whuber
    Oct 30 '17 at 17:27
  • $\begingroup$ @whuber Thanks for the comment. If I may ask a follow-up question, how would the variance-covariance matrix help me figure out the bounds? If you can point me out to some good tutorials or excerpts from books, I'd be very grateful. :) $\endgroup$
    – Milos
    Oct 30 '17 at 17:58
  • 2
    $\begingroup$ All you need to know about any Normal distribution is its mean and variance; the bounds you refer to will depend on the variance. The variance-covariance matrix of the $n^2$ variables $I(x_i\lt y_j)$ determines the variance of $T$. When $n$ is large enough to make $T$ approximately Normal, then you have all the information you need. $\endgroup$
    – whuber
    Oct 30 '17 at 18:03

The statistic $T$ is an example of a general $U$-statistic, introduced by Hoeffding in his 1948 paper A class of statistics with asymptotically normal distribution. Moreover, it is among the most famous of that class, namely the Mann-Whitney U test. It has lower mean squared error than $S$, for it equals $E\left(S\mid X_{\left(1\right)},X_{\left(2\right)},\ldots,Y_{\left(1\right)},Y_{\left(2\right)},\ldots\right)$, and since the order statistics are sufficient, the Rao-Blackwell theorem can be applied. Furthermore, it is normally distributed, a fact that follows from a general theorem on $U$-statistics, see the chapters on $U$-statistics in e.g. Lehmann's Elements of Large Sample Theory or Serfling's Approximation Theorems of Mathematical Statistics. Its limiting distribution, with a small sample correction, can be found on wikipedia., and it is implemented in R through the function $\mathtt{wilcox.test}$ (with the option $\mathtt{paired=FALSE}$).

Note that the definition of the Mann-Whitney U test is ambiguous. Sometimes it is defined as the $T$ above, but sometimes it counts "victories for $x_i$" as positive and "victories for $y_i$" as negative. This is the approach taken in e.g. the R function $\mathtt{wilcox.test}$.

  • $\begingroup$ I font think it's ambiguous, but it can be defined in terms of one condition or its complement, depending on which ordering is of interest (e.g., higher scores are better). $\endgroup$ Aug 29 '20 at 21:26
  • $\begingroup$ The confidence interval may be calculated using a logit transformation to ensure that the bounds remain in the interval [0,1]. $\endgroup$ Aug 29 '20 at 21:28

Here is a fairly straightforward and intuitive argument for both estimators being asymptotically normal.

Call $$T(X|y_1)=\frac{1}{n} \sum_{i=1}^{n} I(x_i>y_1)$$

As with $S$, central limit theorem implies this is asymptotically normal.

Furthermore, take $$T = \frac{1}{n}\sum_{i=1}^{n} T(X|y_i)$$

Since each $T(X|y_i)$ is asymptotically normal, and any linear combination of normal variables is normal, $T$ must too be asymptotically normal.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.