5
$\begingroup$

I was referred to this site from mathoverflow. After some brief searching, I haven't seen another thread on my particular question, but if it's already been answered elsewhere then I hope you'll forgive the spam.

Has anyone ever "ranked" the various commonly-used parametric and non-parametric hypothesis tests in terms of the "strength" of the conclusions that one can ascertain from them? That is to say, if I have two populations that are wildly different from each other (and about which I have absolutely no assumptions whatsoever on the prior distributions), and I'd like to make the strongest possible case that they are different, what would be the right hypothesis test to use?

An example would be if I have one population of samples that all lie between $0$ and $1$ and another population of samples that all lie between $10$ and $11$. Running a $t$-test would likely show that my sample means are different, but I would think that I could make stronger conclusions than that (based on the fact that largest element in one set is smaller than the smallest set in the other, for example).

(as an aside, in the particular case I'm dealing with, one population has lots of samples and the other is quite small, and both have rather small standard deviations)

$\endgroup$
2
  • 3
    $\begingroup$ I find your example a little confusing, John. It refers to a situation with extremely strong distributional assumptions, but it immediately follows a remark about having "absolutely no assumptions whatsoever on the prior distributions," which sounds like exactly the opposite situation. Which is the situation you really have in mind? Also, it is not clear what you mean by "strength." Initially it would seem to refer to force of evidence, but in the example it instead seems to mean the degree of detail (such as going beyond just comparing means). Could you make that idea more precise? $\endgroup$
    – whuber
    Commented Dec 30, 2011 at 22:38
  • 1
    $\begingroup$ Thanks, whuber -- I should have made that more clear. I edited the example to better reflect the question I asked. I'll give careful thought to what exactly I meant by "strength" and edit the question accordingly in the (very) near future. $\endgroup$ Commented Dec 30, 2011 at 23:37

3 Answers 3

3
$\begingroup$

You write

An example would be if I have one population of samples that all lie between 0 and 1 and another population of samples that all lie between 10 and 11.

I am not sure what you mean by "population of samples" but if one population varies between 0 and 1 and the other between 10 and 11, you don't need any hypothesis test. But if you wanted one, a t-test might be OK - the assumptions of a t-test do NOT include that the means are close. They do include equal variances, but there are corrections for that (e.g. Satterthwaite). However, t-tests are not robust to a combination of very different variances and very different sample sizes.

$\endgroup$
0
$\begingroup$

Virtually all parametric statistics make assumptions about underlying distributions and thus are not appropriate. What you want to investigate is robust statistics. Another area of possible use is using the Monte Carlo method, but this is not a hypothesis test per-se, rather a way to test a hypothesis.

$\endgroup$
0
$\begingroup$

If one Distribution provides values in [0, 1] the other in [10, 11] a unique (sample size=1) observation is sufficient to find out what Distribution is. As it was noted a t-test (even if it is possible) is completely superfluous.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.