# How to measure the "confidence" that one set of values is greater than another?

I have a relatively simple task. I have collected several hundred samples from several groups (group here is arbitrary and not relevant to my question). These samples are scored on a continuous interval [0,100]. I know that on average the samples that I collect from some groups are larger than others. I would like to somehow express my "confidence" that one group has larger scores than another. I will illustrate this with an example.

Suppose I am weighing a group of ten oranges and ten apples and find that the weights of my samples are

$$apples = [1,2,3,1,2,3,1,2,3,1]$$ $$oranges = [2,3,4,2,3,4,2,3,4,2]$$

On average the oranges weigh more than the apples in my example. However, not every orange weighs more than all of the apples. How might I express the idea, and confidence of it, that oranges weigh more than apples? Notice that I am not making any assumptions about the distribution of weight for either apples or oranges; they could in fact have very different distributions (i.e. one group may have a normal distribution while another is bimodal). I also do not care about the probability that an orange weighs more than $x$ pounds than an apple; only that oranges are heavier. Is the right way to look at this to find the probability that an orange weights more than an apple, or is it sufficient to say that oranges are heavier on average than apples (saying this worries me since I'm not assuming they have the same distribution)?

Edit: The number of samples I have collected for each group is not equal

• Talking about the probability that an orange weighs more than an apple suggests the Mann-Whitney-Wilcoxon test. But note that if you don't want to make $any$ assumptions about distributions, it's possible that the probability that an apple weighs more than a orange is greater than one half, the probability that a pear weighs more than an apple is greater than one half, and that the probability that an orange weighs more than a pear is greater than one half. Mar 16, 2015 at 17:49
• Regarding @Scortchi's comment: nontransitive dice Mar 18, 2015 at 17:45

Given two sets $A, B$, the average weights have nothing to do with the probability $p$ that $a \in A$ is greater than $b\in B$. $p$ could be almost 100% despite $E(A) < E(B)$ given one huge outlier.

The only ways to do this are to either calculate the true probability by looking at all pairs (like Tusell's suggestion, but all pairs not just non-overlapping pairs) or making some assumption about the distributions of $A, B$ and then estimating hyperparameters. If you assume normality, then the estimated difference is (noncentral, scaled) student's T distributed hence the T test suggested by Mihir.

A one-sided t-test between the two distributions will give you a statistic that expresses (relative to a chosen significance level) whether the average orange is heavier than the average apple.

You can, for example, do this in R:

> a <- c(1,2,3,1,2,3,1,2,3,1)
> o <- c(2,3,4,2,3,4,2,3,4,2)
> t.test(o,a,alternative="greater")

Welch Two Sample t-test

data:  o and a
t = 2.5538, df = 18, p-value = 0.009971
alternative hypothesis: true difference in means is greater than 0
95 percent confidence interval:
0.3209788       Inf
sample estimates:
mean of x mean of y
2.9       1.9


This does assume a t-distribution, but as you increase your sample size, this will generally be a good assumption.

Sure enough, oranges are heavier than apples -- the confidence interval of the mean does not include 0 (which would mean no difference).

• This assumes normality. Mar 16, 2015 at 21:26

Suppose you have two groups with equal number of apples ($A$) and oranges ($O$). You could form $D_i = A_i - O_i$ and then compute $\overline{D}$ and its variance. How far is $\overline{D}$ from 0 in terms of its standard deviation tells how how "different" the two sets are.

If the two groups are not equal in size, you might do as above, pairing elements of the smallest group with elements taken at random fron the largest. This is wasteful, as it discards some observations of the largest.

It seems to me that what you have is a sort of Behrens-Fisher problem, only harder, so likely no answer can be shown optimal.