Here is a MWE of my problem:

I measure the size, $S$, of 10 red apples and 32 green apples.

$\bar S_\mathrm{red} = 8 \pm 1\,\mathrm{cm}$ and $\bar S_\mathrm{green} = 4 \pm 2\,\mathrm{cm}$.

I want to claim that red apples are bigger than green apples, but just reporting the means doesn't feel right because there is a distribution of sizes for each colour.

Ideally, I want to say something like "Red apples are bigger than green apples with 95% certainty."

I was going to just Monte Carlo it, taking a few thousand samples randomly from each and seeing how frequently red is bigger than green, but I feel there is some test for what I'm after.

  • 1
    $\begingroup$ Anyone else feeling Wilcoxon-Mann-Whitney U for this? $\endgroup$ – Dave Jan 23 at 17:39
  • 1
    $\begingroup$ @Dave - I agree, but if all he has are the summary statistics, he's stuck with a $t$ test. $\endgroup$ – jbowman Jan 23 at 17:41
  • 1
    $\begingroup$ If you are really interested in determining if one object is better than another, and you can use large samples or assume normality with small samples, why not simply use $volume$ as your metric for size? If you found significant difference, you could certainly claim evidence to support one object being "bigger" than the other. You could perform tests separately for colors or use regression analysis to control for multiple colors or other factors at the same time. Contrasts would help here. $\endgroup$ – StatsStudent Jan 23 at 20:52

You can resort to the bootstrap version of Student's t-test.

It works as follows:

  • Compute the sample mean and standard deviation for each group and label the results $X_1$ and $s_1$ for group 1, and $X_2$ and $s_2$ for group 2. Set $d_1=\frac{s_1^2}{n_1}$ and $d_2=\frac{s_2^2}{n_2}$, where $n_1$ and $n_2$ are the sample sizes.
  • Generate a bootstrap sample for the first group, compute the sample mean and standard deviation, and label the results $\bar{X_{1}}^{*}$ and $s_{1}^{*}$. Do the same for the second group. Note $d_{1}^{*}$ and $d_{2}^{*}$ accordingly.
  • Compute $$ W^{*} = \frac{(\bar{X_{1}}^{*}-\bar{X_{2}}^{*})-(\bar{X_{1}}-\bar{X_{2}})}{\sqrt{d_1^2+d_2^2}}.$$
  • Repeat Steps 2 and $B$ times ($B$=1000, for example).
  • Put the $W_1^{*},...,W_B^{*}$ in ascending order, yielding $W_{(1)}^{*},...,W_{(B)}^{*}$.
  • Set $L=\frac{\alpha}{2}B$ and $U=(1-\frac{\alpha}{2})B$ and round each of them to the nearest integer.
  • The bootstrap t confidence interval for $\mu_1-\mu_2$ is $$ \Big[ (\bar{X_{1}}-\bar{X_{2}}) + W_{(L)}^{*} \sqrt{d_1+d_2}, (\bar{X_{1}}-\bar{X_{2}})+W_{(U)}^{*} \sqrt{d_1+d_2} \Big] .$$

Assuming that group $1$ corresponds to red apples and group $2$ to green apples, you reject the null hypothesis if and only if $0$ is not located within the calculated confidence interval.

I used the presentation from Chapter $6$ in

Wilcox, R. (2010). Fundamentals of Modern Statistical Methods: Substantially Improving Power and Accuracy. Springer Science & Business Media.

  • $\begingroup$ Doesn't this use the same null hypothesis as the usual t-test? $\endgroup$ – Dave Jan 23 at 19:10
  • 1
    $\begingroup$ Yes but it does not assume the samples are Gaussian and it works well for small sample sizes (i.e. < 30) unlike the classical t-test. $\endgroup$ – Mickybo Yakari Jan 23 at 20:20
  • $\begingroup$ The issue seems to be about limiting inference to a difference in means, which is what the t-test (and therefore this method) does. $\endgroup$ – Dave Jan 23 at 20:32
  • $\begingroup$ @MickyboYakari Regarding your comment about the classical $t$-test, you may want to study the answers to this question. The $t$-test was specifically developed for small samples with the very first application with a sample size of four! $\endgroup$ – COOLSerdash Jan 23 at 21:05
  • 1
    $\begingroup$ I'll revise my answer to add appropriate caveats once I have brushed up on the bootstrap again. $\endgroup$ – Mickybo Yakari Jan 24 at 11:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.