So I have a dataset I've split into training & test with an 80/20 ratio. It's a hieararchically structured dataset, individual human participants each with many observations, so when splitting I ensure that there's an 80/20 split within each participant. I fit to the training data via (excuse my idiosyncratic code-style):
fit = mgcv::bam(
formula = power ~
(
te(
lat , long , cov
, d = c(2,1)
, bs = c("sos","cr")
, k = c(32,10) #max supported by num unique
)
+ te(
lat , long , cov, participant
, d = c(2,1,1)
, bs = c("sos","cr","re")
, k = c(32,10,40) #max supported by num unique
)
)
, data = training_data
, method = "fREML"
, discrete = TRUE
, chunk.size = 0
, nthreads = parallel::detectCores()
, gc.level = 0
)
Which seems to converge just fine. Then I get the predictions for both the training (as a sanity-check) and testing datasets, once with predictions sensitive to the term that includes Participant as a random effect, and once with predictions sensitive only to the group-level terms, then compute the prediction error for each and finally compute the ratio of said prediction errors to the sum original variance (computed by hand as a sanity check):
(
both_data #has both training and testing datasets, indicated by column `dataset`
%>% dplyr::mutate(
re_term_present =
(
mgcv::predict.bam(
fit
, newdata = both_data
, discrete = T
, n.threads = parallel::detectCores()
)
)
, re_term_absent =
(
mgcv::predict.bam(
fit
, newdata = both_data
, discrete = T
, n.threads = parallel::detectCores()
, exclude = 'te(lat,long,cov,participant)'
)
)
)
%>% tidyr::pivot_longer(
cols = c(re_term_present,re_term_absent)
, names_to = 're_term'
, values_to = 'model_prediction'
)
%>% dplyr::group_by(
dataset
, re_term
)
%>% dplyr::mutate(
model_squared_error = (power-model_prediction)^2
, null_squared_error = (power-mean(power))^2
)
%>% dplyr::summarise(
model_sse = sum(model_squared_error)
, null_sse = sum(null_squared_error)
, model_null_ratio = model_sse/null_sse
, prop_accounted = 1-model_null_ratio
)
)
Yielding the table:
dataset re_term model_sse null_sse model_null_ratio prop_accounted
<chr> <chr> <dbl> <dbl> <dbl> <dbl>
1 test re_term_absent 139259. 105291. 1.32 -0.323
2 test re_term_present 86585. 105291. 0.822 0.178
3 train re_term_absent 590409. 454435. 1.30 -0.299
4 train re_term_present 300710. 454435. 0.662 0.338
The result for predictions that take into account the random effects of participants (re_term_present
) make sense: predictions for both test and training account for a positive proportion of the variance of the observed data, with test predictions achieving a lower proportion-accounted-for than the training predictions.
But I can't wrap my head around the result for predictions that don't take into account the random effects of participants (re_term_absent
); how can the model predictions be adding variance? I thought the term that includes Participants as a random effect would be modelled as deviations centered on the mean-across-participants term, in which case I'd expect maybe as low as 0% variance accounted-for if said mean term were indeed zero, but certainly not negative values. How am I thinking about this incorrectly?
te()
bases contain a tonne or redundant terms because the first three covariates are the same. This could well be part of the problem so I'd suggest decomposing into "main" effects and "interaction" withti()
:power ~ te(lat, long, cov) + s(participant, bs = 're') + ti(lat, long, cov, participant)
where I've not repeated your basis specifications except to indicate the "main" effects of theparticipant
random effect. $\endgroup$ti()
doesn't contain the basis functions from the "main" effect term, thete()
in my formula, while your 4-termte()
repeats the basis functions from the 3-termte()
in your formula. $\endgroup$