In my research I have run into the following general problem: I have two distributions $P$ and $Q$ over the same domain, and a large (but finite) number of samples from those distributions. Samples are independently and identically distributed from one of these two distributions (though the distributions may be related: for example, $Q$ may be a mixture of $P$ and some other distribution.) The null hypothesis is that samples come from $P$, alternate hypothesis is that samples come from $Q$.
I am trying to characterize the Type I and Type II errors in testing the sample, knowing the distributions $P$ and $Q$. Particularly, I am interested in bounding one error given the other, in addition to the knowledge of $P$ and $Q$.
I have asked a question on math.SE regarding the relationship of Total Variation distance between $P$ and $Q$ to hypothesis testing, and received an answer that I accepted. That answer makes sense, but I still have not been able to wrap my mind around the deeper meaning behind the relationship of Total Variation distance and hypothesis testing as it relates to my problem. Thus, I decided to turn to this forum.
My first question is: is total variation bound on the sum of the probabilities of Type I and Type II errors independent of the hypothesis testing method that one employs? In essence, as long as there is a non-zero probability that the sample could have been generated by either one of the distributions, the probability of at least one of the errors must be non-zero. Basically, you can not escape the possibility that your hypothesis tester will make a mistake, no matter how much signal processing you do. And Total Variation bounds that exact possibility. Is my understanding correct?
There is also another relationship between Type I and II errors and the underlying probability distributions $P$ and $Q$: the KL divergence. Thus, my second question is: is KL-divergence bound only applicable to one specific hypothesis testing method (it seems to come up around the log-likelyhood ratio method a lot) or can one apply it generally across all hypothesis testing methods? If it's applicable across all hypothesis testing methods, than why does it seem to be so very different from the Total Variation bound? Does it behave differently?
And my underlying question is: is there a prescribed set of circumstances when I should use either bound, or is it purely a matter of convenience? When should the result derived using one bound hold using the other?
I apologize if these questions are trivial. I am a computer scientist (so this seems like a fancy pattern matching problem to me :) .) I know information theory reasonably well, and have graduate background in probability theory as well. However, I am just starting to learn all of this hypothesis testing stuff. If needed, I will do my best to clarify my questions.