Using the distance of p-value from alpha

I have two statistical tests which are inverse of each other, meaning that the null hypothesis are reversed. I want to use both the tests to take a decision. For this purpose, I am planning to do the following:

• If both the tests point to same result (by, say, rejecting the null hypothesis in test A and not rejecting the null hypothesis in test B, or vice versa), then I go ahead and take that decision
• On the other hand, if the results of the two tests are conflicting, I measure the difference of the p-values from alpha (say, 0.05) for each test, and go with the one having the largest deviation (I have some tie-breaking rules, but let's leave that here)

It sounds reasonable to me, but is there some statistical ground in interpreting p-values like this? I certainly haven't seen similar application before (in my limited exposure).

Edit: Let me clarify the question some more with the actual context and tests. I am testing for unit roots in time series, in order to determine if the series needs to be differenced in order to render it mean-stationary. The particular tests are KPSS test, with null being no unit roots, and ADF test with null being an unit root exists. Although the null hypotheses are related (and inverse of each other), but the test regression and the statistics are quite different in my opinion.

-
Instead of controlling type I error while minimizing type II you can minimize the sum of type and type II error (or any combination)... in this case you don't use the likelihood ratio like in Neymann pearson but it's as easy (looks like bayes rule with uniform prior). Anyway, recall that under the null your p-value is uniform (can be close to 0.05) also, I don't really like your procedure. –  robin girard Nov 2 '10 at 17:12
Could you clarify what it means for null hypotheses to be "reversed"? E.g., if one null is that two means are equal, do you take the other null to be that they are unequal? (This particular example leads to trouble...) –  whuber Nov 2 '10 at 18:08

I think you are confusing the concepts of stationarity vs unit roots. Unit root implies non-stationarity but the converse need not hold. Thus, KPSS and ADF tests are not both a test for unit roots. KPSS tests for stationarity whereas ADF tests for one particular form of non-stationarity (i.e., existence of unit roots).

I would suggest going through these notes and in particular see the figure 1 excerpted from the notes below for example of non-stationary series that are not because of unit roots:

-
@Srikant: Thanks for the link to the notes. I am actually trying to detect (D) [I have separate tests for (B) and (C) which I run before (D)] and from that perspective it does seem like one test is reverse of other. Other than that, do you also have a comment about the procedure itself that I was thinking of using? –  Samik R Nov 2 '10 at 23:19

I have a question based on what you asked and assuming I understood you correctly.

Why do you need two null hypotheses to decide about one decision?

Are you doing this just because we null hypotheses can only be rejected and never accepted?

Now the answer to your question about using p-value as a measure of "truth of the null" (similar to measuring the distance of p-value from $\alpha$) is not justified from a statistical perspective. This can lead to possible wrong decisions. Here is a great reference explaining the dangers of using p-values in the sense you are trying to use.

-
Thanks for your comment. The reason why I want to be using the result from both the test is that, I have some evidence that by choosing a particular null hypothesis, the test result is actually getting biased towards what the null hypothesis tests. Thanks for the paper link though - will go through it. –  Samik R Nov 2 '10 at 17:26
@Samik in a way, when you do a hypothesis test using p-values, you are assuming the null to true and thus the bias for the null. Its just a suggestion: try using Bayes Factor, may be that will remove the "biasness" you observe (although, I must remark, its not a guarantee that this will work) –  suncoolsu Nov 2 '10 at 17:38
thanks, will read up on that. I have also added the actual context of the question in the OP. –  Samik R Nov 2 '10 at 18:19
Hi, can you suggest some reference for Bayes Factor that you mentioned? Thanks. –  Samik R Nov 19 '10 at 16:52
@Samik. Check this out bit.ly/9YXqCy –  suncoolsu Nov 19 '10 at 17:38