7
$\begingroup$

On page 146 of Gelman's Bayesian Data Analysis, Gelman discusses Bayesian p-value as a way to check the fit of the model. The idea is to compare the observed data ($y$) with data that could have been generated by the model if we replicate the experiment ($y^{rep}$).

He defines Bayesian p-value as

$$ p_B = Pr(T(y^{rep}, \theta) \geq T(y, \theta) | y) $$

I don't quite understand why it makes sense to have the test statistic be a function of the parameters, $\theta$. Indeed, if the goal is to "compare the observed data with data that could have been generated by the model", shouldn't the comparison be strictly between $y$ and $y^{rep}$?

For example, on the same page, Gelman provides an example where he checks the fit of a normal model. The test statistic is:

$$ T(y, \theta) = | y_{(61)} - \theta | - |y_{(6)} - \theta | $$

where $\theta$ is the mean of the normal model. This test statistic is designed to ignore the model fit at the extreme tail, beyond the 6th and 61th order statistics.

Why don't we use the following test statistic instead, relying purely on the data?

$$ T(y, \theta) = | y_{(61)} - \bar y | - |y_{(6)} - \bar y | $$

$\endgroup$
0
$\begingroup$

There is nothing stopping you from using test statistics based solely on the replicate data.

The point in the example is that the model assumes $y_i$ is normally distributed around a mean $\theta$ not around $\overline{y}$. Thus the test statistic provided in Gelman et. al. tests something about the normality assumption whereas its not really clear what your test statistic is testing.

$\endgroup$
  • 3
    $\begingroup$ I agree that it is 100% kosher to use test statistics based solely on the replicate data. The question is why is it kosher to use test statistics that involve the model parameters? Conceptually, it makes sense that we use the model to make some predictions, then compare those predictions against observed "ground truth" $T(y)$. It does not make sense that the comparison can be made against $T(y, \theta)$, which involves $\theta$, which comes from the model that we want to check in the first place. It seems circular to me. $\endgroup$ – Heisenberg May 20 '16 at 14:43

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.