In a recent task at my job I have come with a situation I cannot find references to research made about it. Basically, I have two procedures and I want to test if they behave exactly the same way (i.e., they will always give the same response when fed with the same input).

More precisely, say I have procedure 1 which, when fed with an input, will give a sequence as a response, say ABAAB. This procedure is the industry standard, but it is expensive.

Now, there is a (cheaper) proposed procedure 2 which should behave exactly the same way. So, if for three consecutive and (maybe) dependent inputs procedure 1 outputs ABCAA, ACAAB and CABBA; I should see the same three sequences as output from procedure 2.

Lets call $X_n$ the Bernoulli variable which equals 1 if both procedures give the same answer and 0 otherwise. So $X_n \sim Ber(p)$ and my null hypothesis is $p = 1$.

I cannot figure how to test this hypothesis with a finite sample. I can easily propose tests for the hypothesis $p > 1 - \epsilon$, but I would like to check if there is some work done regarding the original problem. Thank you in advance.

  • $\begingroup$ An obvious rejection rule is to reject as soon as you see any difference whatever between the two sets of results. If the null is true, you will never reject, so the type I error rate is 0, the lowest it can be, but if the null is false, you will expect some difference to pop up eventually, so a large sample should often have excellent power (we can't say what it is without more information, though) $\endgroup$ – Glen_b Aug 13 '18 at 5:21
  • $\begingroup$ @glen_b Thank you for your comment. I know that as soon as I see some $X_m = 0$ then $H_0$ is to be rejected. But I have a finite sample (say a couple of thousand inputs) so my question is what confidence do I have with this test. Because if I see all the two thousands realizations equal one, under the alternative hypothesis the probability of this outcome is arbitrarily close to one if $p = 1 - \epsilon$ and $\epsilon$ is sufficiently small. $\endgroup$ – Cristián Antuña Aug 13 '18 at 10:27
  • $\begingroup$ Just as I explained when discussing power above, we can't do anything specific (like derive a 5% test say) with the present information; you mention the possibility of dependence but we don't know anything about the form or the extent of it. $\endgroup$ – Glen_b Aug 13 '18 at 10:54
  • $\begingroup$ @glen_b Thank you again for your comment. I thought this was the situation (nothing more specific can be done unless more information is addedd) but it was worth asking in case it was not and I was unaware of it. I guess my best chances are to production department to give me more specific hypothesis to work with. Thanks again! $\endgroup$ – Cristián Antuña Aug 13 '18 at 11:03

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.