Using DIC to test which data should be used - different sample sizes

Question

I have a glmm model

$y \sim b_1 * b_2 * b_3 + random$

where $b_i$ are the fixed effects. I am using DIC to compare models and select the best fitting model. I also have some options in setting up my model. For example, $b_1$ is the average spring temperature in the individuals first 5 years of life. This can be calculated in two ways:

Approach 1 - stringent:

Average temperature is calculated only in cases where all five years have data, such that if an individual was born in the year $x_0$, we would need information on temperature for years $x_0$ through $x_4$ to get the 5 year average.

Approach 2 - relaxed:

Average temperature is calculated for all cases where at least one of the 5 years has data recorded. Therefore, if we had the temperature for years $x_0$, $x_1$ and $x_4$ the score for $x_0$ individuals would be $\frac{(x_0 + x_1 + x_4)}{3}$, and for $x_1$ individuals $\frac{(x_1 + x_4)}{2}$ etc..

Can I use DIC based model selection to choose approach? The random effects and all other model structure is the same, it's just a case of substituting fixed effect $b_1$. Because the data is more stringently selected in approach 1 it greatly reduces the sample size, but it does mean the samples are more precise: does the different sample size in either model have impacts on my decision to use DIC based model selection methods?

---

Judging from the answer on my previous question I suspect the answer is that I can make this comparison. However, it would be really excellent to get a citable source on this.

---

As another illustration, imagine I have data from two populations, with measures taken over many years. Greater sampling effort was given to one population, such that there is a tendency that one population has a smaller sample size, and even has no samples some years. Could I compare models using both populations and just the well sampled data to see if the poor balance in experimental design is affecting model fit? Could I select the model with data only from the well sampled population on this basis? Does experimental balance affect DIC?

Answer 1

The short answer is no: the DIC comparison is not valid between these models applied to fundamentally different datasets.

The slightly longer answer is that DIC (like all likelihood-based methods of model comparison such as AIC) is based partly on the -2 * log likelihood of the model - as you add more data points this likelihood will decrease, so the DIC will increase (for a fixed number of effective parameters) regardless of how well the model 'fits' in the general sense. So in your case, the 'stringent' approach would be expected to give a lower DIC purely on the basis of having fewer observations to fit to than the 'relaxed' approach model. If you wanted to persuade yourself of this, you could simulate a number of datasets of different N, and then fit the exact same model to these and look at the relationship between DIC and N.

The second part of the following question is somewhat related to yours and the answer may also help you:

Comparing AIC among models with different amounts of data

You could potentially use non-likelihood based model selection techniques, such as posterior predictive p-values or leave one out cross validation, to see if there was evidence for the 'less certain' datapoints having worse fit to the model. However, I would be very reluctant to combine averages calculated using a different number of datapoints within the same model from a theoretical point of view, as this would likely break assumptions of homoscedasticity. It would probably be better to fit something like a hierarchical model of the individually observed temperatures with a random effect of individual.

Regarding the last part of your question - it depends what you mean. Experimental balance will certainly affect the fit of the model, in that a better balanced study design will give less uncertainty in results. This would be expected to increase the diagnostic power of DIC to differentiate between candidate models (fit to the same data). If you mean 'would a well balanced study have a better DIC than an equivalent but unbalanced study' then the answer is that the DIC would not be comparable, as the datasets are by definition not the same.