Why does adding a lag effect increase mean deviance in a Bayesian hierarchical model? - Cross Validated most recent 30 from stats.stackexchange.com 2019-08-17T21:14:18Z https://stats.stackexchange.com/feeds/question/143098 http://www.creativecommons.org/licenses/by-sa/3.0/rdf https://stats.stackexchange.com/q/143098 14 Why does adding a lag effect increase mean deviance in a Bayesian hierarchical model? Jeromy Anglim https://stats.stackexchange.com/users/183 2015-03-24T01:38:33Z 2015-03-29T06:14:15Z <p><strong>Background:</strong> I'm currently doing some work comparing various Bayesian hierarchical models. The data $y_{ij}$ are numeric measures of well-being for participant $i$ and time $j$. I have around 1000 participants and 5 to 10 observations per participant.</p> <p>Like with most longitudinal datasets, I am expecting to see some form of auto-correlation whereby observations that are closer in time have a greater correlation than those that are further apart. Simplifying a few things, the basic model is as follows:</p> <p>$$y_{ij} \sim N(\mu_{ij}, \sigma^2)$$</p> <p>where I am comparing a no lag model:</p> <p>$$\mu_{ij} = \beta_{0i}$$</p> <p>with a lag model:</p> <p>$$\mu_{ij} = \beta_{0i} + \beta_{1} (y_{i(j-1)} - \beta_{0i})$$</p> <p>where $\beta_{0i}$ is a person-level mean and $\beta_1$ is the lag parameter (i.e., the lag effect adds a multiple of the deviation of the observation from the previous time point from the predicted value of that time point). I've also had to do a few things to estimate $y_{i0}$ (i.e., observation prior to the first observation). </p> <p>The results I am getting indicate that:</p> <ul> <li>The lag parameter is around .18, 95% CI [ .14, .21]. I.e., it's non-zero</li> <li>Mean deviance and the DIC both increase by several hundred when the lag is included in the model</li> <li>Posterior predictive checks show that by including the lag effect, the model is better able to recover the auto-correlation in the data</li> </ul> <p>So in summary, the non-zero lag parameter and the posterior predictive checks suggest the lag model is better; yet mean deviance and DIC suggest that the no lag model is better. This puzzles me.</p> <p>My general experience is that if you add a useful parameter it should at least reduce the mean deviance (even if after a complexity penalty the DIC is not improved). Furthermore, a value of zero for the lag parameter would achieve the same deviance as the no lag model.</p> <h3>Question</h3> <p><strong>Why might adding a lag effect increase mean deviance in a Bayesian hierarchical model even when the lag parameter is non zero and it improves posterior predictive checks?</strong></p> <h3>Initial thoughts</h3> <ul> <li>I've done a lot of <strong>convergence checks</strong> (e.g., looking at traceplots; examining variation in deviance results across chains and across runs) and both models seem to have converged on the posterior.</li> <li>I've done a code check where I forced the lag effect to be zero, and this did recover the no lag model deviances.</li> <li>I also looked at mean deviance minus the penalty which should yield deviance at expected values, and these also made the lag model appear worse.</li> <li>Perhaps the lag effect reduces the effective number of observations per person which reduces the certainty in estimating the person level means ($\beta_{0i}$) which increases deviance.</li> <li>Perhaps there is some issue with how I've estimated the implied time point before the first observation.</li> <li>Perhaps the lag effect is just weak in this data</li> <li>I tried estimating the model using a maximum liklihood using <code>lme</code> with <code>correlation=corAR1()</code>. The estimate of the lag parameter was very similar. In this case the lag model had a larger log likelihood and a smaller AIC (by about 100) than one without a lag (i.e., it suggested the lag model was better). So this reinforced the idea that adding the lag should also lower the deviance in the Bayesian model.</li> <li>Perhaps there is something special about Bayesian residuals. If the lag model uses the difference between predicted and actual y at the previous time point, then this quantity is going to be uncertain. Thus, the lag effect will be operating over a credible interval of such residual values.</li> </ul> https://stats.stackexchange.com/questions/143098/-/143892#143892 1 Answer by Summit for Why does adding a lag effect increase mean deviance in a Bayesian hierarchical model? Summit https://stats.stackexchange.com/users/61648 2015-03-29T06:14:15Z 2015-03-29T06:14:15Z <p>Here are my thoughts:</p> <ul> <li>Instead of DIC, BIC, AIC I suggest to directly work with the <em>marginal likelihood</em> (also known as <em>evidence</em>) if you can afford it. The larger the <em>evidence</em>, the more likely is your model class. It may not make a large difference, but DIC, BIC, AIC are, after all, only approximations.</li> <li>In order to check if a lag-effect leads to a larger <em>marginal likelihood</em>, I suggest to perform the following initial check: Take the model that includes the lag-parameter. (a) Fix the lag-parameter to $0.18$. (b) Set the lag-parameter to <em>zero</em>. Compute the <em>marginal likelihood</em> of both model classes. Model class (a) should have the larger <em>marginal likelihood</em>. </li> <li><p>Let's go a step further: Take the model that does not consider the lag-effect (c) and compute its <em>marginal likelihood</em>. Next, take your model class (d) that incorporates the lag-effect and has a prior on the lag-parameter; compute the <em>marginal likelihood</em> of (d). You would expect that (d) has a larger <em>marginal likelihood</em>. So what, if you don't?: </p> <p>(1) The <em>marginal likelihood</em> considers the model class as a whole. This includes the lag-effect, the number of parameters, the likelihood, the prior. </p> <p>(2) Comparing models that have a different number of parameters is always delicate, if there is considerable uncertainty in the prior of the additional parameters.</p> <p>(3) If you specify the uncertainty in the prior of your lag-parameter unreasonably large, you penalize the entire model class. </p> <p>(4) What is the information that supports equal probabilities for negative lags and for a positive lag? I believe that it is very unlikely to observe a negative lag, and this should be incorporated in the prior. </p> <p>(5) The prior that you chose on your lag-parameter is uniform. This is usually never a good choice: Are you absolutely sure that your parameters must really lie inside the specified bounds? Does each lag-value inside the bounds really have equal likelihood? My suggestion: go with a beta-distribution (if you are sure that the lag is bounded; or with the log-normal if you can exclude values smaller than <em>zero</em>.</p> <p>(6) This is a particular example, where the use of <em>non-informative</em> priors is not good (looking at the <em>marginal likelihood</em>): You will always favor the model that has a smaller number of uncertain parameters; it does not matter how good or bad the model with more parameters could do. </p></li> </ul> <p>I hope my thoughts give you some new ideas, hints?!</p>