4
$\begingroup$

I can't decide what is the best way to assess the stability of a higher order fractional polynomial model. To use an example I have been working on, I am analyzing a dataset with panel data selected from 5 different universities. I am modelling the non-linear changes of bone mass during lifetime. Using F-test and log-likelihood tests the results suggest that the best-fitting model consists of three powers, Age^3, Age^2 and a linear component of Age. I am then fitting the selected model in Stata:

fp <Age>, center fp(3 2 1) replace: reg Bone_Mass <Age> i.Sex i.Uni1 i.Uni2 i.Uni3 /*      
*/ i.Uni4 i.Uni5
estimates store eq_1
predict fit1

Sex and Uni are entered as dummy variables in the model. Because this identified model is more complex than regression models with one or two powers I wanted to some further verification that it is actually a stable model. Research has extensively described the mfpboot command to resample residuals and make sure about variable inclusion in multivariate fractional polynomial models (e.g. here and here). However, I am actually fitting a univariate model.

My first idea was to leave a cohort out and try to replicate the model selected. However, if all cohorts have a significant effect on the best-fitting model why would I be expecting to replicate the solution? Another idea would be to actually bootstrap the residuals for the identified regression using the separate powers as predictors and see whether they all remain significant in this model. Something that looks like this for example:

gen Age3=Age^3
gen Age2=Age^2
bootstrap, reps(1000): reg Bone_Mass Age*_b[Age_1] Age2_b*[Age_2] Age3*_b[Age_3]/*   
*/i.Sex i.Uni1 i.Uni2 i.Uni3 i.Uni4 i.Uni5

where _b[x] represent the regression coefficients from the final fitted fractional polynomial model

I don't, however, see how multicollinearity would not cause issue in the interpretation of the standard errors of these predictors.

$\endgroup$
1
$\begingroup$

It seems that to me that your route to choosing this kind of model using fractional polynomial methodology is secondary. As I understand it you are essentially fitting a family of cubics in age, the family being distinguished by sex and universities. Why the latter should matter in predicting bone mass is an intriguing detail. (I am puzzled why the whole exercise is regarded as a univariate model.)

I've grown to distrust cubics, which often give a pessimal combination of moderate fit, incorrect limiting behaviour and lack of theoretical interpretation.

I'd be most worried about overfitting, especially getting unbiological artefacts, at worst prediction of negative values and more possibly being wrong about the number and location of turning points and inflexions. Presumably there is prior knowledge which guides on what is sensible.

So, I might bootstrap data and use a composite graph as sample statistic to look for instability in the pathological sense of any curves being qualitatively wrong.

$\endgroup$
10
  • $\begingroup$ What I am primarily interested in is the non-linear change of bone-mass during the life-time. I agree that fractional polynomials might not be optimal way to identify significant turning points and the age at which bone mass peaks, however it's a very intuitive method and you don't make any a priori assumptions about the distribution of the dependent variable like you would with GMM for example. Sex and universities are just being adjusted for and are therefore of secondary interest. In fact, the produced graph and the fitted line are very much in line with what was $\endgroup$
    – StevenP
    Mar 12 '15 at 14:09
  • $\begingroup$ expected. In the last part of your answer, do you imply I should bootstrap the polynomial terms like I described in the question? Thank you very much for taking the time $\endgroup$
    – StevenP
    Mar 12 '15 at 14:09
  • $\begingroup$ I am not sure I understand your specific bootstrap suggestion, but it seems to me that it presumes the major point at issue, whether cubics are a good idea. I never thought of GMM as an alternative here! Splines and local polynomials would be top of my list; it's not as if the coefficients of cubics have any useful interpretation, or is it? $\endgroup$
    – Nick Cox
    Mar 12 '15 at 14:40
  • $\begingroup$ I thought about local polynomials too but I didn't want to make any a priori assumptions about where to place the knots. Moreover, I read a paper by Binder et al. (2013) (ncbi.nlm.nih.gov/pubmed/23034770) suggesting that fp are better than splines to identify simpler functions and that they are pretty much comparable. The point I was trying to make which was probably not very clear is if the whole model including all powers of Age (i.e. Age^3, Age^2 and Age) is stable enough, in terms of not over-fitting the data and it being replicable. Therefore, a permutation technique $\endgroup$
    – StevenP
    Mar 12 '15 at 15:37
  • 1
    $\begingroup$ You are using Stata: lpoly requires no specification of knots. $\endgroup$
    – Nick Cox
    Mar 12 '15 at 15:42

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.