0
$\begingroup$

So I have a dataset I've split into training & test with an 80/20 ratio. It's a hieararchically structured dataset, individual human participants each with many observations, so when splitting I ensure that there's an 80/20 split within each participant. I fit to the training data via (excuse my idiosyncratic code-style):

fit = mgcv::bam(
    formula = power ~ 
        (
            te(
                lat , long , cov
                , d = c(2,1)
                , bs = c("sos","cr")
                , k = c(32,10) #max supported by num unique
            ) 
            + te(
                lat , long , cov, participant
                , d = c(2,1,1)
                , bs = c("sos","cr","re")
                , k = c(32,10,40) #max supported by num unique
            )
        )
    , data = training_data
    , method = "fREML"
    , discrete = TRUE
    , chunk.size = 0
    , nthreads = parallel::detectCores()
    , gc.level = 0
)

Which seems to converge just fine. Then I get the predictions for both the training (as a sanity-check) and testing datasets, once with predictions sensitive to the term that includes Participant as a random effect, and once with predictions sensitive only to the group-level terms, then compute the prediction error for each and finally compute the ratio of said prediction errors to the sum original variance (computed by hand as a sanity check):

(
    both_data #has both training and testing datasets, indicated by column `dataset`
    %>% dplyr::mutate(
        re_term_present = 
            (
                mgcv::predict.bam(
                    fit
                    , newdata = both_data
                    , discrete = T
                    , n.threads = parallel::detectCores()
                )
            )
        , re_term_absent = 
            (
                mgcv::predict.bam(
                    fit
                    , newdata = both_data
                    , discrete = T
                    , n.threads = parallel::detectCores()
                    , exclude = 'te(lat,long,cov,participant)'
                )
            )
    )
    %>% tidyr::pivot_longer(
        cols = c(re_term_present,re_term_absent)
        , names_to = 're_term'
        , values_to = 'model_prediction'
    )
    %>% dplyr::group_by(
        dataset
        , re_term
    )
    %>% dplyr::mutate(
        model_squared_error = (power-model_prediction)^2
        , null_squared_error = (power-mean(power))^2
    )
    %>% dplyr::summarise(
        model_sse = sum(model_squared_error)
        , null_sse = sum(null_squared_error)
        , model_null_ratio = model_sse/null_sse
        , prop_accounted = 1-model_null_ratio
    )
)

Yielding the table:

  dataset re_term         model_sse null_sse model_null_ratio prop_accounted
  <chr>   <chr>               <dbl>    <dbl>            <dbl>          <dbl>
1 test    re_term_absent    139259.  105291.            1.32          -0.323
2 test    re_term_present    86585.  105291.            0.822          0.178
3 train   re_term_absent    590409.  454435.            1.30          -0.299
4 train   re_term_present   300710.  454435.            0.662          0.338

The result for predictions that take into account the random effects of participants (re_term_present) make sense: predictions for both test and training account for a positive proportion of the variance of the observed data, with test predictions achieving a lower proportion-accounted-for than the training predictions.

But I can't wrap my head around the result for predictions that don't take into account the random effects of participants (re_term_absent); how can the model predictions be adding variance? I thought the term that includes Participants as a random effect would be modelled as deviations centered on the mean-across-participants term, in which case I'd expect maybe as low as 0% variance accounted-for if said mean term were indeed zero, but certainly not negative values. How am I thinking about this incorrectly?

$\endgroup$
3
  • 2
    $\begingroup$ I wouldn't fit the model the way you are doing; the two te() bases contain a tonne or redundant terms because the first three covariates are the same. This could well be part of the problem so I'd suggest decomposing into "main" effects and "interaction" with ti(): power ~ te(lat, long, cov) + s(participant, bs = 're') + ti(lat, long, cov, participant) where I've not repeated your basis specifications except to indicate the "main" effects of the participant random effect. $\endgroup$ Commented Jan 27, 2021 at 19:45
  • 2
    $\begingroup$ The key here is that the ti() doesn't contain the basis functions from the "main" effect term, the te() in my formula, while your 4-term te() repeats the basis functions from the 3-term te() in your formula. $\endgroup$ Commented Jan 27, 2021 at 19:45
  • $\begingroup$ Thanks! I’ll give that a try and report back in a few hours when the results are in. $\endgroup$ Commented Jan 27, 2021 at 21:03

0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.