Suppose I have two models $M_1$ and $M_2$ and I want to compare their performances in terms of measure (accuracy on classification instances, i.e. number of correct predictions to size of all instances). I conjecture that the difference in their results is not significant. So

  • Null H: $M_1$ and $M_2$ are different (in terms of accuracy measure).
  • Alternative H: $M_1$ and $M_2$ are not different (in terms of accuracy measure).

First: Is this hypothesis testable? Is it is well-defined?

I have seen the reverse scenario in different places. For example, the paired-permutation test in [1] is designed for a different hypothesis (where null is not-difference of the two predictors).

At the link they're comparing generalization accuracies (the response) for two models. The test statistic is the mean difference in generalization accuracy. Under the null that the models have equal mean accuracy (and the additional assumption that they have the same distribution under the null), the model labels that go with the pairs of accuracies are arbitrary -- you could interchange them (flipping the sign of the difference in accuracy) without altering the distribution of differences.

If in the original algorithm in [1] I change the condition from $$ |\mu_{new}| \geq |\mu_{old}| $$ to $$ |\mu_{old}| \leq |\mu_{new}| $$ Second: Is it a valid algorithm for my defined hypothesis testing?

Third: If not, any suggestion on how to test this hypothesis?

[1] http://axon.cs.byu.edu/Dan/478/assignments/permutation_test.php


closed as unclear what you're asking by whuber Jan 27 '15 at 0:11

Please clarify your specific problem or add additional details to highlight exactly what you need. As it's currently written, it’s hard to tell exactly what you're asking. See the How to Ask page for help clarifying this question. If this question can be reworded to fit the rules in the help center, please edit the question.

  • 1
    $\begingroup$ You should carefully define the terms you use. What do $M$ and $\mu$ represent? I suspect "$\mu$ is being used here in a way that would be likely to be misunderstood given the usual statistical conventions. $\endgroup$ – Glen_b Jan 26 '15 at 22:46
  • $\begingroup$ See the link I have included in the question. $\endgroup$ – Daniel Jan 26 '15 at 22:48
  • 1
    $\begingroup$ Simply referring to the link for definitions is not sufficient: $\:$ (1) Later readers should not have to read the link simply to understand the question. (2) In any case, a post must have enough context to stand alone, since links can die without warning; this is explained in the help. Links for additional context are fine, but you can't rely on them continuing to be available. (3) I don't think the link sufficiently explains the situation; some background is absent. While your edits have definitely improved the question substantially, please clarify further in your question. $\endgroup$ – Glen_b Jan 26 '15 at 23:24
  • 1
    $\begingroup$ The current title ("Permutation test: when the null hypothesis is") seems to be missing something from the end? $\endgroup$ – Silverfish Jan 26 '15 at 23:56
  • 1
    $\begingroup$ Well, they're still not specific enough to actually find a null distribution -- "in terms of a desired measure" doesn't pin down the measure. I believe it's clear enough to you which measure is intended, I think it's better to just state it and reduce the opportunity for people to think it's still unclear in at least that respect. $\endgroup$ – Glen_b Jan 27 '15 at 5:38

Hypotheses should usually be phrased as a statement about population quantities (though in some cases, possibly infinite-dimensional), and that statement should generally be a null statement (typically about lack of difference); in particular, it needs to be a statement under which a null distribution of a test statistic can be computed (or at least an edge-case null can).

In the case of a permutation test, you need that the permutations are equally likely; the null needs to make the labels (or whatever it is that's being 'swapped around') at least exchangeable. Yours does not.

Assuming I've correctly guessed at the meaning of some of the symbols, the two expressions of null hypotheses in the first few lines of your link are suitable null hypotheses (though I'd probably phrase them slightly differently, they're clear enough and should be fine).

You say:

The pair-permutation test in [1] is designed for a different hypothesis (where null is non-significance of the two predictors).

This is how null hypotheses have to work; if you make "difference" the null (an "open" compound hypothesis), the difficulty is in finding the distribution of the test statistic under the null; the limiting case would actually be in the alternative. In the case of a permutation test, this means that the permutation could only be done under the alternative you give... so that, perforce, must be the null hypothesis.

It might be that your needs might be better served by some form of equivalence test, but this may be tricky in a permutation test context.

[Note that you can, however, flip the direction of a one-sided test if you need to (between $\leq$ and $\geq$), as long as it definitely includes the equality (the edge-case under which you can compute the distribution of the null).]

  • $\begingroup$ Rephrased the question. $\endgroup$ – Daniel Jan 26 '15 at 22:48
  • $\begingroup$ Thanks, that helps. I'll have to make some edits to my answer. $\endgroup$ – Glen_b Jan 26 '15 at 23:25

Not the answer you're looking for? Browse other questions tagged or ask your own question.