Let's say I have a sample A: [0,0,0,1] and another sample B: [2,0,5,10,100,3,2,6]
I would like to know the probability that A and B are both picked from the same population C.
I tried applying a hypothesis test, but it gives me a p value of approx. 0.39 and I think it should be clear that it's very unlikely that both samples are from the same distribution.
You don't say what kind of hypothesis test you used. Doing inference on such small samples as these is always going to be difficult.
However, assuming that these are two independent random samples, a nonparametric Kolmogorov-Smirnov test (in R) does reject the null hypothesis that these two samples were randomly sampled from the same population.
There is a warning message that (on account of the ties), the P-value is not exact, but 0.034 seems sufficiently smaller than 0.05 to say that we can reject at the 5% level.
x1 = c(0,0,0,1)
x2 = c(2,0,5,10,100,3,2,6)
ks.test(x1, x2)
Two-sample Kolmogorov-Smirnov test
data: x1 and x2
D = 0.875, p-value = 0.0337
alternative hypothesis: two-sided
Warning message:
In ks.test(x1, x2) : cannot compute exact p-value with ties
Similar data without ties gives a 'cleaner' test--rejecting the null hypothesis with no warning messages.
y1 = c(.01, .02, .03, .9)
y2 = c(2,0,5,10,100,3,2.1,6)
ks.test(y1, y2)
Two-sample Kolmogorov-Smirnov test
data: y1 and y2
D = 0.875, p-value = 0.0202
alternative hypothesis: two-sided
Another possible test is the two-sample Wilcoxon (rank sum) test. Its distribution theory is also somewhat disturbed by ties, but it does find a significant difference between your two samples. Looking just at the P-value, we have:
wilcox.test(x1,x2)$p.val
[1] 0.02434338
Warning message:
In wilcox.test.default(x1, x2) :
cannot compute exact p-value with ties
