3
$\begingroup$

Is there a way to get a prediction from an gam model (from package mgcv) that contains random effect smooths, where the new data contains a level of the random effect that didn't exist in the training set? Something comparable to the "re.form=NA" parameter in lme4:predict.merMod ?

library(hflights) # get a big data set
hflights$UniqueCarrier <- factor(hflights$UniqueCarrier) # turn carrier into a factor
newhflights <- head(hflights) # create "new" d.f. for prediction
newhflights$UniqueCarrier <- "ZZ" # change carrier to a previously unseen value 


library(mgcv)
m_gam <- gam(ActualElapsedTime ~ s(Distance) + s(TaxiOut, UniqueCarrier, bs="re"), data=hflights)
predict(m_gam, newdata=newhflights) # fails with "ZZ not in original fit"

library(lme4)
m_lmer <- lmer(ActualElapsedTime ~ Distance + (TaxiOut | UniqueCarrier), data=hflights)
predict(m_lmer, newdata=newhflights) # fails with "new levels detected in newdata"
predict(m_lmer, newdata=newhflights, re.form=NA) # works

I have searched for an answer and found this question, but it didn't appear to be directly related to my problem. Predicting with random effects in mgcv gam But I did try adding a dummy variable to the model based on it.

hflights$dummy <- 1
newhflights$dummy <- 0
m_gam <- gam(ActualElapsedTime ~ s(Distance) + s(TaxiOut, UniqueCarrier, bs="re", by=dummy), data=hflights)
predict(m_gam, newdata=newhflights) # still doesn't work, not in original fit

I'm out of ideas. Am I overlooking something obvious?

Improve this question
$\endgroup$
3
  • 1
    $\begingroup$ I suppose refitting the model with ZZ added as a level to the original data is out of the question? (you'd need to refit with drop.unused.levels = FALSE). Alternatively, just set that new level to be one of the existing levels; as long as you've turned off the re term it will make no difference to the predictions (assuming the models are as per the ones shown). $\endgroup$
    – Gavin Simpson
    Commented Jun 12, 2015 at 20:16
  • $\begingroup$ The new previously unseen levels are not necessarily known until prediction time, and refitting the model is not a desirable option. Overwriting the random effect level with a known one (then changing it back post-prediction) seems to work, and I guess that's the only real option. It seemed awkward to do it that way, so I wondered if there was a more elegant solution. Thanks! $\endgroup$
    – colonel.triq
    Commented Jun 15, 2015 at 21:42
  • $\begingroup$ No, even Simon Wood freely admits that handling of random effects is simple at best in mgcv::gam. Perhaps mgcv::gamm or gamm4::gamm might work better for you seeing as the spline fixed-effects part of your model appears relatively simple. $\endgroup$
    – Gavin Simpson
    Commented Jun 16, 2015 at 2:17

0

Reset to default

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.