In Doing Bayesian Data Analysis (link to the book) and Bayesian Estimation Supersedes the t-Test, J. Kruschke proposes using the following criterion to reject or accept the null hypothesis in a location experiment:

  • Accept null hypothesis if the 95% HDI falls completely inside the ROPE
  • Reject the null hypothesis if the 95% HDI falls completely outisde the ROPE

Presumably ROPE is given by a desired effect size (correct me if I am wrong).

Is there a way to connect the above decision rule with the optimization of a posterior expected loss? If so, what would is the loss function that corresponds to this criterion?

  • 1
    $\begingroup$ Is there a specification for when the HDI is only partially overlapping with the ROPE? $\endgroup$ – user44764 Jun 12 '14 at 14:20
  • $\begingroup$ @Matthew I think the specification for that case is "inconclusive". $\endgroup$ – Amelio Vazquez-Reina Jun 12 '14 at 15:47

There is no way to phrase this problem in terms of posterior expected loss in a useful way. The 95% HDI is completely determined by the model and the data, and has no variance (i.e., it is a constant). The ROPE is pre-specified, so it also has no variance (i.e., it is also a constant). Let's pretend that they are random variables. Because they have no variance and are thus constant, this means that in every case $$ P(\text{HDI}_{0.95} \cap \text{ROPE} = \emptyset \mid \text{data}) \in \{ 0, 1 \}. $$

The problem comes in when you try to incorporate the loss function. For now let's assume that accept and inconclusive are the same result (you stick with the null). Since there are only two possible cases (the HDI is completely in or not completely in the ROPE), we must use a loss function that only varies with the decision to accept or reject. Let's suppose you have a penalty $w_I$ for the Type I error of rejecting the null incorrectly and a penalty $w_{II}$ for the Type II error of not rejecting when you should have. There is $0$ loss in the case that you are correct.

The expected loss for each decision is $$ EL[\text{accept}] = w_{II} \times P(\text{HDI}_{0.95} \cap \text{ROPE} = \emptyset \mid \text{data}), \\ EL[\text{reject}] = w_{I} \times P(\text{HDI}_{0.95} \cap \text{ROPE} \ne \emptyset \mid \text{data}). $$ In the case that the HDI is completely in the ROPE, $EL[\text{accept}] < EL[\text{reject}]$ regardless of the values of $w_{I}, w_{II}$, and vice versa. So the specification of a loss function really gets you no where you weren't already.

The purpose of using the HDI is really to outflank the usual necessity of specifying the loss function. For example, a more natural question might be to ask what is $$ P(\theta \in \text{ROPE} \mid \text{data}). $$ Then we might make a table of values like before (specifying $w_{I}, w_{II}$ and $0$ for being correct). In which case $$ EL[\text{accept}] = w_{II} \times P(\theta \notin \text{ROPE} \mid \text{data}), \\ EL[\text{reject}] = w_{I} \times P(\theta \in \text{ROPE} \mid \text{data}). $$ We would reject when (assuming all nonzero) $$ \frac{EL[\text{reject}]}{EL[\text{accept}]} > 1 \\ \iff \frac{ w_{I} \times P(\theta \in \text{ROPE} \mid \text{data})}{w_{II} \times P(\theta \notin \text{ROPE} \mid \text{data})} > 1 \\ \iff \frac{ P(\theta \in \text{ROPE} \mid \text{data})}{P(\theta \notin \text{ROPE} \mid \text{data})} > \frac{w_{II}}{w_{I}}. $$


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.