14
$\begingroup$

When using natural (i.e. restricted) cubic splines, the basis functions created are highly collinear, and when used in a regression seem to produce very high VIF (variance inflation factor) statistics, signaling multicollinearity. When one is considering the case of a model for prediction purposes, is this an issue? It seems like it will always be the case because of the nature of the spline construction.

Here is an example in R:

library(caret)
library(Hmisc)
library(car)
data(GermanCredit)

spl_mat<-rcspline.eval(GermanCredit$Amount,  nk=5, inclx=TRUE) #natural cubic splines with 5 knots

class<-ifelse(GermanCredit$Class=='Bad',1,0) #binary target variable
dat<-data.frame(cbind(spl_mat,class))

cor(spl_mat)

OUTPUT:
              x                              
    x 1.0000000 0.9386463 0.9270723 0.9109491
      0.9386463 1.0000000 0.9994380 0.9969515
      0.9270723 0.9994380 1.0000000 0.9989905
      0.9109491 0.9969515 0.9989905 1.0000000


mod<-glm(class~.,data=dat,family=binomial()) #model

vif(mod) #massively high

OUTPUT:
x         V2         V3         V4 
319.573 204655.833 415308.187  45042.675

UPDATE:

I reached out to Dr. Harrell, the author of Hmisc package in R (and others) and he responded that as long as the algorithm converges (e.g. the logistic regression) and the standard errors have not exploded (as Maarten said below) - and the model fits well, best shown on a test set, then there is no issue with this collinearity.

Further, he stated (and this is present on page 65 of his excellent Regression Modeling Strategies book) that collinearity between variables constructed in an algebraic fashion like restricted cubic splines is not an issue as multicollinearity only matters when that collinearity changes from sample to sample.

Improve this question
$\endgroup$
1
  • 1
    $\begingroup$ You can always orthogonalize the generated splines (for example the rcsgen Stata command uses Gram-Schmidt orthogonalizaton) $\endgroup$
    – boscovich
    Commented Sep 25, 2013 at 21:12

1 Answer 1

Reset to default
8
$\begingroup$

The multicollinearity can lead to numerical problems when estimating such a function. This is why some use B-splines (or variations on that theme) instead of restricted cubic splines. So, I tend to see restricted cubic splines as one potentially usefull tool in a larger toolbox.

Improve this answer
$\endgroup$
7
  • $\begingroup$ Hi Maarten, when you say numerical problems do you refer to convergence or such? If the regression converged and appears to do well on a test set - do you conclude the situation is not a problem? $\endgroup$
    – B_Miner
    Commented Sep 25, 2013 at 11:17
  • $\begingroup$ If there are numerical problems then lack of convergence is a likely (but not a necessary) consequence. Unrealistic coefficient estimates and/or unrealisticly high standard errors are other possible consequences. $\endgroup$
    – Maarten Buis
    Commented Sep 25, 2013 at 11:37
  • 1
    $\begingroup$ Methods like QR decomposition helped a lot. However, you can still break modern software, it has just become harder to do so. $\endgroup$
    – Maarten Buis
    Commented Sep 26, 2013 at 10:21
  • 1
    $\begingroup$ That may not be a problem. But if there is a numerical problem, the scale=TRUE argument to some of the fitting functions in the R rms package will standardize all columns before fitting and unstandardize coefficients after fitting in a way that the user doesn't need to account for. $\endgroup$
    – Frank Harrell
    Commented Mar 20, 2017 at 15:15
  • 2
    $\begingroup$ Don't look at individiual coefficients and standard errors so much. Look at the entire fitted curve. $\endgroup$
    – Frank Harrell
    Commented Mar 21, 2017 at 12:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.