I'm currently trying to get lifetables for our population based on death and population counts. My first idea was to follow this paper methodology but after some discussions we are planning to use different models, such as GLMMs and GAMs and explore the inclusion of different knots and families.

The author suggested to use train and test sets or cross-validation. I've been reading (great resources and questions on this site!) and I would like to know if the methodology I'm using is appropriate as I'm not entirely sure.

Here's what I have in mind:

1. Shuffle the data
2. Divide the data into training (80%) and test set (20%).
3. Test the different parameters in the training data, and compare them using the test data.
4. Once I have the chosen family and knots, shuffle the data again.
5. Do n iterations (20?) of 5-fold cross validation (size dataset=3440 observations) of one model to get the performance of the methodology (The idea is that the model chosen in 4 is the one able to better predict the data).

Now, here are my questions and confusion:

Should I use a nested cross-validation and in the inner loop test the model parameters? Even though I want to try several parameters? The theoretical number of knots may be too low or I may need to add a smooth factor to one of the variables, etc. I'm not sure if I can play with these parameters during cross-validation.

I'm also not entirely sure about cross-validation to compare methodologies. Do I need to do one k-fold cross validation per method? If I use cross-validation for a GAM with a poisson family and a GAM with a negative binomial I may be on a k-fold where that model is able to better predict the data, where in k-1-fold the same model could not be as pro-efficient.

My other question relates to the output parameters and how to evaluate the cross validation. Since my aim is to predict I was planning to look at the residuals of each model, AIC, and plot the predicted and observed values to see which method performed better. Should I use different parameters?

I appreciate any help or guidance as I just started reading about cross validation and have no experience.

Thank you!