If the true populations of A and B have mean(A) > mean(B), and my one-tailed significance test in a sample incorrectly indicates that mean(B) > mean(A). Is this a type I error, type II or both?
$\begingroup$
$\endgroup$
5
-
$\begingroup$ This reads rather like a routine textbook-style question (as might be asked in coursework for example). But in any case: are you aware of the definitions of type I and type II error? $\endgroup$– Glen_bCommented Dec 18, 2015 at 22:32
-
$\begingroup$ No. I'm not taking any classes. This is a real data problem I'm facing, and I tried to post it as a MWE. I understand type I and type II errors in the two-tailed case, but not much in the one-tailed case. In the example I gave, I suspect this is worse than both types of errors. So a direct answer or a pointer to materials that explain a similar phenomenon will be appreciated. $\endgroup$– mossaabCommented Dec 19, 2015 at 4:17
-
$\begingroup$ In what real data problem do you need to know which error type it is? Can you explain the context in which knowing the name of the error would be important? $\endgroup$– Glen_bCommented Dec 19, 2015 at 4:33
-
$\begingroup$ I have an evaluation measure that can (correctly) compare the performance of two search engines on 1000 queries. I'm experimenting with (new) approximated evaluation measures that can be used when we don't have enough information about the relevance of the documents returned by the search engines. These new evaluation measures produce errors (such as the one I'm referring to in this question). My goal is to (meta)evaluate these evaluation measures in terms of actual significance level and power, so that people can make an informed decision about using them (when no alternative exists). $\endgroup$– mossaabCommented Dec 19, 2015 at 11:49
-
$\begingroup$ That would be very useful context in your question. $\endgroup$– Glen_bCommented Dec 19, 2015 at 15:25
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
6
Let's simply look at some definitions:
a type I error is the incorrect rejection of a true null hypothesis
a type II error is the failure to reject a false null hypothesis
(If you don't refer to the definitions when faced with a question like this, you're wasting your time.)
Clearly, and directly from those definitions:
You cannot commit a type II error if you reject (you either correctly reject when $H_0$ is false, or you commit a type I error, by rejecting a true null).
Similarly, you cannot commit a type I error if you fail to reject (you either fail to reject when you shouldn't reject, or you commit a type II error by failing to reject a false null).
If the true populations of A and B have mean(A) > mean(B),
i.e. if that reality is your null*
* (presumably you'll need to include equality in the null as well)
and my one-tailed significance test in a sample incorrectly indicates that mean(B) > mean(A).
and it incorrectly rejects that null,
Is this a type I error, type II or both?
it cannot be type II since it involves a rejection. If your null was in the same direction as reality (but not in the same direction as the sample) then it's rejecting incorrectly, so it's a type I error.
-
$\begingroup$ I (already) understand from the definition that if I incorrectly state that mean(A) = mean(B), then this is a type II error; and that if I incorrectly state that mean(A) != mean(B), then this is a type I error. $\endgroup$– mossaabCommented Dec 19, 2015 at 11:55
-
$\begingroup$ @mossaab That would be the case for a two-tailed test. Can you state the null and alternative for the one-tailed test? $\endgroup$– Glen_bCommented Dec 19, 2015 at 13:55
-
$\begingroup$ Thanks for taking the initiative of formulating mean(A) > mean(B) as the null, which begins to clarify things. Both mean(B) < mean(A) and mean(A) = mean(B) are incorrect alternatives. That is, I don't want to include the equality in the null, but falling in the incorrect inequality is worse than falling in the equality. My understanding from your edited answer is that the fact than one case is worse than the other is irrelevant to the type of error, and then they both fall into type I error. Correct? $\endgroup$– mossaabCommented Dec 19, 2015 at 17:27
-
$\begingroup$ 1. You have to include the equality case in the null, I'm afraid. It's not something you can really avoid (you need the limit of the interval to be in the interval, because that's actually where you will calculate your type I error rate at -- the point at which the rate is largest). If that inclusion of equality is unacceptable, you probably need to think more deeply about the situation; you may not have a hypothesis testing problem at all. $\,$ 2. Sorry, I really didn't follow what you were saying at the end there. Both of what fall into type I error? You only asked about one error in your Q $\endgroup$– Glen_bCommented Dec 19, 2015 at 22:29
-
$\begingroup$ (2) is related to (1), I guess. The true population has mean(A) > mean(B), if in the sample I see that mean(A) = mean(B), then this is an error (and all what I'm looking for is a type name). If the sample has mean(B) > mean(A), this is a different (and actually a worse error), for which I'm also looking for a name. I cannot have the equality in the null, because I already know it is not. It appears that hypothesis testing is not something I can use to model my problem. $\endgroup$– mossaabCommented Dec 20, 2015 at 1:57