3
$\begingroup$

I've run 4 models (simple LM, quadratic model, GLMM, and GLMM with quadratic) to predict tree age (age) from tree diameter (D) for each of 42 species (SPEC). The diameter data has all been log transformed to account for non-normality and heavy weighting of near-zero values. I've compared each of these models using AIC separately for each of the species and chose the model with the lowest AIC to use for each species.

I have a rough idea about what tree age predictions should look like (i.e., not super large or negative). The problem is that most of my 'best' models (generally the GLMM with quadratic) make essentially nonsense predictions for that species. However, if I look at the predictions of 'worse' models (typically the simple LM) for those given species, the predicted age values make more sense. So I would be more inclined to use these 'worse' models in these instances. The 'worse' model that is the best predictor is further not always consistent between species and some species' ages are best predicted from the model with the lowest AIC -- in other words, it's not consistent.

My question is: Is there a methodological way to choose the model post hoc regardless of how it ranks in terms of AIC (and instead by how it predicts sensible values with new data)?

Improve this question
$\endgroup$
2
  • $\begingroup$ I will admit that many of the discrepancies occur with trees with large D, of which few or none exist in the training data. $\endgroup$
    – theforestecologist
    Commented Jul 20, 2015 at 17:54
  • 1
    $\begingroup$ I wonder if any of the models you have tried include interactions between species and other predictors. If not, you're assuming the same relationship for all species, with only a scale factor for each species. That sounds likely to be very limiting. It is true that interactions with species will create a lot of extra parameters - but maybe you can divide species into a small number of groups (based on biological criteria, preferably) and use interactions with the grouping factor instead. $\endgroup$
    – Russ Lenth
    Commented Jul 21, 2015 at 18:52

1 Answer 1

Reset to default
3
$\begingroup$

It sounds like you're looking for cross-validation. In many contexts, it's used to tune the parameters of a model when the primary goal is prediction, using a process summarized like so:

  1. Select a subset of the available data, say 80%, and use it to train the model.
  2. Select the parameter value that minimizes prediction error on the remaining, unseen subset.

There are diverse implementations of this technique, but that's the overall gist. You can also apply the same principle for model selection: Simply choose the model that minimizes prediction error on unseen data.

Improve this answer
$\endgroup$
3
  • $\begingroup$ What if my data set is too small to split the data into training data? (because I assume I need to know the actual values I'm trying to predict from the test the data in order to use them to determine prediction error). $\endgroup$
    – theforestecologist
    Commented Jul 20, 2015 at 18:14
  • $\begingroup$ Also, as I commented above, many of the issues I see in predicted values occur with trees with large D, of which few or none exist in the training data. (For example, the largest tree in the training data is D=25, but the largest tree needing a predicted value is D=75). Can cross-validation help with this? $\endgroup$
    – theforestecologist
    Commented Jul 20, 2015 at 18:16
  • 1
    $\begingroup$ @theforestecologist both good questions. For the first, I'd search the cross-validation tag for questions about CV on small datasets. You may also want to look for questions about ensuring that a test set is representative. To the second question, my intuition is that no, one cannot count on CV error estimates to generalize well outside the range of inputs used to train the model. But justifying that formally is a little above my pay grade :) in any case, I'd suggest asking separate questions on these specific topics. $\endgroup$
    – Sean Easter
    Commented Jul 20, 2015 at 21:18

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.