1
$\begingroup$

Let us say we draw 10 sample values from a given (continuous) population. However, the values tend to be very different than expected, so that we suspect that there might have been an error with the measurement method. Therefore, we draw again 10 values in seemingly the same way. Yet, the values of the second sample tend to be similar.

Is it now justifiable to pool the two samples to a single sample with a sample size of 20, and if so, how exactly can this be justified?

Let us say we use a T- or a Mann-Whitney-U-Test in order to test whether there are significant differences between the two samples. The result is that the two samples do not differ significantly. Yet, even if we cannot reject the null hypothesis, this not a proof that the two sample come indeed from the same population (and can be pooled).

Instead, one could conduct an equivalence test (parametric or non-parametric). If we can reject the null-hypothesis, we were able to show that the average differences between the sample are within a certain bound. Yet, strictly speaking this is not a proof either, since there is always an error probability. I have learned that statistical tests can never fully proof that two samples come from the same distribtuion. So one could make the case that statistical testing is the wrong approach here.

However, to me it seems also hard to deny that the fact that we were able to show that the two samples are singnificantly similar increased the confidence in the correctness of the measurements. Could statistical testing somehow help to justify pooling the two samples?

$\endgroup$
1
  • 1
    $\begingroup$ It sounds as if you are not sure of your sampling method. Until you can be more specific about the difference between the two samples, it may be difficult to know what kinds of tests might be applicable. Of course, any two samples of size 20 will have some superficial differences--even if your sampling method is correct. $\endgroup$
    – BruceET
    Oct 15, 2021 at 5:06

1 Answer 1

0
$\begingroup$

If we look at a well-defined problem with specific data, then I can illustrate how to a two-sample Wilcoxon test and a two-sample t test in R, and give brief interpretations. Based on my comment, I am not sure the following matches your actual situation.

Suppose you have two truly random samples of size $n = 20$ from the same population, $\mathsf{Gamma}(\mathrm{shape}=5,\mathrm{rate}-0.1),$ as in R below.

set.seed(2021)
x1 = rgamma(20, 5, 0.1)
x2 = rgamma(20, 5, 0.1)
median(x1); median(x2)
[1] 45.55794
[1] 51.10196

Boxplots show distributions of similar shape (slightly right-skewed) with slightly different medians). One has a mild outlier the other has no outliers.

enter image description here

boxplot(x1, x2, horizontal=T, 
  col="skyblue2", names=T, pch=20)

A two sample Wilcoxon rank sum test shows P-value $0.6017 > 0.05 = 5\%,$ so the difference in medians is not surprising in view of the overall dispersion of the data.

wilcox.test(x1, x2)

        Wilcoxon rank sum test

data:  x1 and x2
W = 180, p-value = 0.6017
alternative hypothesis: 
 true location shift is not equal to 0

Normal probability plots seem to show that the data are not consistent with sampling from a normal distribution because they are not close to being linear--especially not the first sample. So it is questionable whether a two-sample t test will give reliable results. (This is not surprising because we are using data simulated from gamma distributions, but in a real application we would not have this information.)

enter image description here

par(mfrow = c(1,2))
 qqnorm(x1, col="blue");  qqline(x1, col="blue")
 qqnorm(x2, col="blue");  qqline(x2, col="blue")
par(mfrow = c(1,1))

A Welch 2-sample t test finds no significant difference between the means of the two samples. I show only the P-value because the validity of the details is in doubt.

t.test(x1, x2)$p.val
[1] 0.5750906

Depending on the data from your two samples, you might use such tests to see if there is a significant difference between the two samples. Bear in mind that two samples of size 20 from the same population may "look different" in superficial ways, but it may be risky to conclude that they are truly different unless a formal test detects a difference.

Even if I did not know that the two samples are taken from the same gamma distribution, I see nothing in the formal tests to say they might be from different distributions.

Note: Because the two samples are of the same size, we could do a Kolmogorov-Smirnov test to see if they are from the same distribution. For samples of size 20 the K-S test does not have very good power to detect a difference, but some statisticians would want to see its results. No significant difference is found.

ks.test(x1, x2)

        Two-sample Kolmogorov-Smirnov test

data:  x1 and x2
D = 0.2, p-value = 0.832
alternative hypothesis: two-sided

The test statistic $D$ is the maximum vertical distance between the empirical CDF (ECDF) of the first sample (blue) and the ECDF of of the second (brown).

enter image description here

plot(ecdf(x2), col="brown", main="ECDF Plots")
 lines(ecdf(x1), col="blue")
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.