1
$\begingroup$

I have a Bayesian model and was looking to do some model checking via Posterior Predictive p-values.

From Bayesian Data Analysis (Gelman et al) it is stated that we adopt a test quantity $T(y,\theta)$ in order to assess whether the statistic with random replications of the data exceeds the statistic from our original data.

$$T(y^{\text{rep }s},\theta^{s})\geq T(y,\theta^{s})$$

Now, it states that the test quantity can either be a function of the data $T(y)$ or a function of both the data and the parameters $T(y,\theta)$.

I can easily think of statistics to assess that are a function of the data (mean, variance etc.). However, I am struggling to think of a statistic that could also depend on the parameters. I think I am obviously overlooking some simple examples.

Improve this question
$\endgroup$

1 Answer 1

Reset to default
1
$\begingroup$

I'm sure there are many answers. Bayesian Data Analysis (3rd edition) page 146 already gives an example:

$$ T(y, \theta) = |y_{(61)} - \theta| - |y_{(6)} - \theta| $$

where $\theta$ is the mean of your normal model. This is from an example with 66 observations, so ordered observations 6 and 61 are chosen to roughly assess the 10% and 90% quantiles. This test quantity evaluates whether the normal model is adequate for the 'main body' of the distribution - the middle excluding the extreme tails.

Another example could be mean squared error in a regression model: $$ T(y, \theta) = \frac{||y - X \theta||^2_2}{n} $$

This also depends on both your data and parameters.

Improve this answer
$\endgroup$
6
  • $\begingroup$ Thanks for your response. I'm confused as my understanding is the data is a function of the parameters i.e. $y=f(\theta)$. So how does it make sense to compare the $y$ with the $\theta$? $\endgroup$
    – statsplease
    May 22, 2018 at 7:43
  • $\begingroup$ I'm not sure I can take away your confusion, but I'll try. I think when you say 'the data is a function of the parameters' this is not quite true - the data arise randomly from a process with certain parameters. We are trying to understand that process and find the parameters that make most sense in the context of the observed data and the process. In this sense the data depend on the parameters but are not a function of those parameters. For example, if flip a coin twice, both observations can be different even though they come from the same process (with identical parameter). Hope this helps $\endgroup$
    – Maurits M
    May 22, 2018 at 7:49
  • $\begingroup$ Sorry, I probably wasn't clear. Take your MSE example. Surely, when we perform the replication (i.e. generating the data using a random parameter(s) from the posterior), the replicated data $y^{\text{rep }s}$ will always be equal to $X\theta^{s}$ (because it was generated from that $\theta^{s}$)? so the MSE for the replicated data will always be zero? $\endgroup$
    – statsplease
    May 22, 2018 at 7:55
  • $\begingroup$ Ah, now I understand where you're going. With regular linear regression you would generate $y^{rep}$ from a normal distribution with mean $X \theta$ and some variance $\sigma$ that you estimated, so $T(y^{rep}, \theta)$ is definitely not always zero. In fact, you should compare this statistic to $T(y, \theta)$ to see if your simulated data behaves similarly to the original data. Then you can formulate alternative tests (like fit only on the left or right tail) to further assess how your model compares with the data. $\endgroup$
    – Maurits M
    May 22, 2018 at 8:08
  • $\begingroup$ Thanks again. Does the above MSE statistic only work in this kind of regression context? What if my model was simply $Y\sim F(y;\theta)$ where $F$ denotes some distribution? I have data $y$ and can sample $\theta^{s}$ which gives me $T(y,\theta^{s})$ but how would I do the same for $T(y^{\text{rep} s},\theta^{s})$? $\endgroup$
    – statsplease
    May 22, 2018 at 8:42

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.