4
$\begingroup$

My problem is the following: I'm evaluating the fit to a function $f(\mathbf{x},\theta)$ via MCMC (because I have some priors on the parameters), and I'm trying to evaluate the DIC, given by:

$$\rm{DIC}=\bar{D}+p_D,\ \ \ \ (1)$$

where, if we define the deviance $D(\theta)=-2\log(\mathcal{L(\theta|\mathbf{x})})$, and $L(\theta|\mathbf{x})$ is my likelihood, then $$\bar{D} = E^{\theta}[D] = -2E^{\theta}[\log(\mathcal{L(\theta|\mathbf{x})})],\ \ \ \ (2)$$ where $E^\theta[\cdot]$ represents the expected value over the posterior distribution, and the efective number of parameters, $p_D$, is given by $$p_D = \bar{D}-D(\hat{\theta})\ \ \ \ (3)$$ where $\hat{\theta}$ is the posterior expectation of the parameters. This is so far what I understood from the paper by Spiegelhalter et al. (2002).

The thing is that, just for testing purposes, I'm using a simple gaussian likelihood of the form: $$\mathcal{L}(\theta|\mathbf{x}) = \frac{1}{(2\pi \sigma^2)^{n/2}}\exp\left(-\sum_{i=1}^n\frac{\mathbf{x}_i-\mathbf{f}_i(\theta)}{2\sigma^2}\right),$$ and I'm getting HUGE values for $p_D$. Considering $\sigma$ as a parameter and the fact that $\theta$ is 12-dimensional in my particular application, I would expect values of $p_D$ close to 13, but I get values in the order of 900! What I'm doing to evaluate everything is the following:

  1. After thinning my MCMC chain, for each link, I get a value of the likelihood. This gives me a sample of the value of the likelihood at each link, and then I estimate $E^\theta[D]$ as: $$\hat{D} = \frac{1}{L}\sum_{i=1}^LD(\theta_i),$$ where $L$ is the number of links and $\theta_i$ is the value of the parameters (including $\sigma$) at that link.
  2. I get the posterior mean of my parameters from all my MCMC links (supposing I'm sampling from the posterior) in order to get $\hat{\theta}$. With estimates for this and for $E^\theta[D]$, I just replace the values in eqs. (2) and (3) to obtain (1).

With this procedure I get reasonable values for my estimate of $E^\theta[D]$, but $p_D$ just doesn't seem right. Is there anything wrong in my procedure?

Thanks in advance for the help!

$\endgroup$

1 Answer 1

3
$\begingroup$

No clear flaw with your implementation. First, I have reservations about using DIC, from using the data twice, to being based on a linear model intuition that works poorly in more complex models like mixtures. So your model may be of that kind. Second DIC is sensitive to parametrisation so using the posterior mean may not be the right choice. Third, if $f$ is highly nonlinear, maybe the effective dimension is more like $n$ than like $p$...

$\endgroup$
5
  • $\begingroup$ Yeah, my model $f$ is highly nonlinear, but I'm somewhat astonished by the (bad?) results of the DIC. AIC and BIC both select models according to my experience, but DIC selects the model that I would never think was the 'correct'/closer 2 the 'correct' one. The problem is that I'm using posterior distributions, and AIC doesn't seem suited for the problem from the beggining; the same goes to BIC. I've looked at other model selection criteria (or extensions of AIC and BIC), but they usually require the computation of Fisher matrices which are intractable in my case...do you have any suggestion? $\endgroup$
    – Néstor
    Apr 30, 2013 at 8:18
  • $\begingroup$ Oh, and, in my case, $n=90$. Getting $p_D\approx 900$ seems like something strange is going on... $\endgroup$
    – Néstor
    Apr 30, 2013 at 8:21
  • $\begingroup$ did you check your code when $f(\theta)=\mu$? $\endgroup$
    – Xi'an
    Apr 30, 2013 at 14:47
  • $\begingroup$ No, I'm performing an MCMC run to check that, thanks for the suggestion. $\endgroup$
    – Néstor
    May 1, 2013 at 6:50
  • $\begingroup$ I never got back to answer my question, but the problem apparently had to do with two parameters that I had. The thing is that in my model, the ratio between them was the observable (say, $\theta_1/\theta_2$) but I foolishly set both parameters separately (say, $\theta_1$ and $\theta_2$). This generated a lot of dispersion in $\theta_1$ and $\theta_2$ because there are an infinite number of possibilities if the ratio is the ('fixed') observable, and this caused the effective number of parameters to explode :-)! Thanks for helping me in checking that everything that I was doing was ok! $\endgroup$
    – Néstor
    Jul 11, 2013 at 4:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.