Sorry for the following basic questions but it is important for me to get a feedback for my approach. I would like to create reference values for children in form of percentile curves (also called Z-Score Curves, example here). For this purpose, I use the GAMLSS package in R and the LMS method that was proposed by Cole (here). The GAMLSS package provides a set of distributions, one of them is the Box-Cox Cole Green distribution.
I have two questions about the fitting and the assessment procedure:
How to find the best value for degree of freedom for the parameters L, M and S?
How to assess the generalization of my chosen model?
Currently, I do it the following way:
1.) Following code fits a GAMLSS model with a Box-Cox Cole Green (BCCG) distribution to the training data. The degree of freedom (df) for the penalized spline functions (pb) for L, M and S is set to 1.
model <- gamlss(y ~ pb(x, df = 1), sigma.formula = ~ pb(x, df = 1), nu.formula = ~pb(x, df = 1), family = "BCCG", method = RS(), data = training_data)
Now, I want to find the best setting for each degree of freedom (df). It is a model selection task, so I use the generalized Akaike information criterion (GAIC) for this decision. The GAMLSS package comes with the find.hyper() function, which conveniently compares models with different df and gives the model with the lowest GAIC as output. This is chosen to be the best model. An interesting question here is whether GAIC is also suitable to compare models with different distributions? Can I compare the GAIC for any model or are there any limitations?
Alternatives to the GAIC would be the Schwarz Bayesian information Criterion (SBC) or cross validation. As I do not have a big number of data, I do not intend to use cross validation. Can you recommend any of the alternatives or other methods?
2.) For the assessment of the generalization of the best model, I separate my data set into training and test data. The training data is used for the fitting procedure like described above. Then, I use the test data and evaluate the global deviance. This is done by using the getTGD() function. However, I would like to get out more from that approach. Do you have any recommendation for the assessment? I found graphical methods like the Q-Q plot or worm plot. Are these recommended?
I have already found a lot of useful information about GAMLSS in following documents: