4
$\begingroup$

It is often stated that the null hypothesis should be written as an equality (see for example here), like for example $\mu_A-\mu_B=0$, while the alternative uses an inequality (for example, $\mu_B\gt\mu_A$). I think that this kind of null hypothesis is called "simple". My question is, why should we always prefer simple null hypotheses?

Suppose for example that I have an existing machine learning code A, and I develop a new code B, based on a different paradigm (maybe A is a neural network, while B is a random forest). I test A and B on some data. I can define an accuracy metric for the codes, thus for each test I can say whether A or B was more accurate. It seems extremely unlikely to expect that A and B could have the same average accuracy on the population from which the data are drawn. After all, they're completely different algorithms. Thus, a NH such as $H_0:p_A=p_B$ seems quite "unnatural" to me. Rather, since I want to know if new code is more accurate than the old one, I would try to falsify the NH that B is no more accurate than A, i.e., $H_0: p_B\le p_A$. Thus, the alternative would be $H_a:p_B\gt p_A$. Why is this kind of null not appropriate for testing?

I guess the issue is that with the "compound" null (is this the correct term?) I cannot easily find a distribution of a test statistics, and thus I may not be able to perform NHST. For example, in the two sample t-test, the null that the mean of the two samples is the same allows me to derive easily the distribution of the test statistics (mean is zero, and standard deviation is derived from sample standard deviation and number of samples). If instead I assume that one mean is larger than the other, then I don't know the parameters of the test statistics distribution. Is this correct?

$\endgroup$
1
  • 6
    $\begingroup$ I'd say that not everyone agrees with the premise of the.title. Some people certainly take that point of view in respect of the usual tests, but some don't. $\endgroup$
    – Glen_b
    Jun 7, 2016 at 3:02

2 Answers 2

6
$\begingroup$

There is nothing wrong with your proposed test. It is possible to derive the sampling distribution of the null with a compound null. What we do, in essence, is use the sampling distribution of the simple null, and if the truth were that $p_B\ll p_A$, then the rate of type I errors would be less than your stated alpha.

What you are getting at is called a non-inferiority test. Here is a non-gated and practitioner-friendly paper:

It may also help to read some of the existing threads on CV that are related to this topic. I have provided somewhat related answers:

You can also read the threads categorized under the tag.


On an unrelated note, you should not use the t-test to compare the accuracy of two classifiers. Since these are correct or incorrect, you would use some method appropriate for binary data, such as the z-test for two proportions. However, since the classifiers will almost certainly be assessed using the same data, McNemar's test should be used (see my answer here: Compare classification performance of two heuristics).

$\endgroup$
1
  • $\begingroup$ Great answer, thanks! I will be sure to read your references. I didn't know about these concepts (noninferiority and equivalence testing). Concerning the McNemar's test, sure, that's the appropriate test in my specific case (even though I often just use the z-test on the difference of proportions, since I have colossal samples). I was just considering two different examples (one being the comparison of accuracy between two classifiers, and other being the two sample t-test for unpaired data). That probably wasn't clear. But thanks for the pointer anyway! $\endgroup$
    – DeltaIV
    Jun 7, 2016 at 7:48
0
$\begingroup$

If $H_0: p_A-p_B=0$ is not rejected with some significance level, then it doesn't imply that $p_A$ is exactly $p_B$ (this is very basic). It means that $0$ is in an interval around your observed test statistic. The two algorithms can be different but they can return values of some chosen loss function that are close from each other. This is reasonable. If you suspect that the algorithm B is at least as good as A's, you can test $H_0:p_A=p_b$ against $H_1: p_A>p_B$. This is what most of staticians would do....

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.