I'm currently trying to get lifetables for our population based on death and population counts. My first idea was to follow this [paper methodology][1] but after some discussions we are planning to use different models, such as GLMMs and GAMs and explore the inclusion of different knots and families.

The author suggested to use train and test sets or cross-validation. I've been reading (great resources and questions on this site!) and I would like to know if the methodology I'm using is appropriate as I'm not entirely sure.

Here's what I have in mind:

 1. Shuffle the data
 2. Divide the data into training (80%) and test set (20%).
 3. Test the different parameters in the training data, and compare them using the test data.
 4. Once I have the chosen family and knots, shuffle the data again.
 5. Do n iterations (20?) of 5-fold cross validation (size dataset=3440 observations) of one model to get the performance of the methodology (The idea is that the model chosen in 4 is the one able to better predict the data).

[![enter image description here][2]][2]

Now, based on the figure I've uploaded, here are my questions and confusion:

**Should I use a nested cross-validation and in the inner loop test the model parameters (as the figure)? Each k-fold correspond to one set of parameters or I need to repeat a nested CV for each set of parameters?** The theoretical number of knots may be too low or I may need to add a smooth factor to one of the variables, etc. I'm not sure if I can play with these parameters in the inner loop/s.

This brings me to my other question. **Do I need to do one k-fold cross validation per method?** If I use cross-validation for a GAM with a poisson family and a GAM with a negative binomial I may be on a fold where that model is able to better predict the data, where in another fold the same model may not perform so well, so instance the set of knots could be inappropriate.  

Another thing that I'm unsure of is that **one variable is calendar year. Do I need special care when shuffling the data? Perhaps make sure each fold has information about all the years or something else?** I also thought of using a GLMM or GAMM, perhaps that would be enough?

My other question relates to the output parameters and how to evaluate the cross validation. Since my aim is to predict I was planning to look at the residuals of each model, AIC, and plot the predicted and observed values to see which method performed better. Should I use different parameters?

I appreciate any help or guidance as I just started reading about cross validation and have no experience.

**EDIT - following @adrin answer**

If I understood your answer, basically nested cross validation would be something along those lines:

    for each fold in the outer loop 
         for each parameter in the grid 
              run each parameter in the inner loop
              evaluate each parameter in each k-fold
         chose best parameter based on results from inner loop
         run best parameter in each k-fold of the outer loop

 The parameters I would like to test are:

1. Different set of knots for age (some fixed at the boundaries but I would like to test a few others for middle ages).

_The reason to choose knots is due to extreme values at the boundaries, as there's a large mortality for the first point (infant mortality) and a large mortality above 85 (since data above 85 is grouped)._ 

_I just got Simon Wood's book so I can understand more about GAMs. I thought about using cubic splines but thin splines could be interesting. It would be interesting to look at the difference between thin plate splines and the use of predefined knots. If thin plate splines fit the data well it would be a great option._

2. GLM with a poisson and NB family and a GAM with poisson and NB family.
3. I would like to check the inclusion of a random effect for region.
4. I would also like to test whether an interaction between gender and age, and region and age improves the model.

_I wasn't thinking about using `ti` for interactions but instead something along those lines:_

    gam(deads ~ region*age + gender*age + calendar_year + ti(age,k=7,bs="cr") + offset(log(pop)), knots=list(age=list_knots), family=poisson, data=data, method="REML")

_But I also thought of trying something like this:_

    gam(deads ~ s(age,bs="fs",by=region) + s(age,bs="fs",by=gender) + calendar_year + ti(age,k=7,bs="cr") + offset(log(pop)), knots=list(age=list_knots), family=poisson, data=data, method="REML")

_Obviously, I need to read more to check what makes sense and is more appropriate._

5. Finally, I was wondering whether I should add some factor smoothing to the factor variable.

Now, perhaps I should have a clearer idea of the final model. Let me hear your thoughts. But having all of these parameters to test was/is some part of my confusion about cross-validation in the inner loop.


Thank you!


  [1]: https://bmcpublichealth.biomedcentral.com/articles/10.1186/s12889-015-2534-3
  [2]: https://i.sstatic.net/OdhVD.png