11
$\begingroup$

When using natural (i.e. restricted) cubic splines, the basis functions created are highly collinear, and when used in a regression seem to produce very high VIF (variance inflation factor) statistics, signaling multicollinearity. When one is considering the case of a model for prediction purposes, is this an issue? It seems like it will always be the case because of the nature of the spline construction.

Here is an example in R:

library(caret)
library(Hmisc)
library(car)
data(GermanCredit)

spl_mat<-rcspline.eval(GermanCredit$Amount,  nk=5, inclx=TRUE) #natural cubic splines with 5 knots

class<-ifelse(GermanCredit$Class=='Bad',1,0) #binary target variable
dat<-data.frame(cbind(spl_mat,class))

cor(spl_mat)

OUTPUT:
              x                              
    x 1.0000000 0.9386463 0.9270723 0.9109491
      0.9386463 1.0000000 0.9994380 0.9969515
      0.9270723 0.9994380 1.0000000 0.9989905
      0.9109491 0.9969515 0.9989905 1.0000000


mod<-glm(class~.,data=dat,family=binomial()) #model

vif(mod) #massively high

OUTPUT:
x         V2         V3         V4 
319.573 204655.833 415308.187  45042.675

UPDATE:

I reached out to Dr. Harrell, the author of Hmisc package in R (and others) and he responded that as long as the algorithm converges (e.g. the logistic regression) and the standard errors have not exploded (as Maarten said below) - and the model fits well, best shown on a test set, then there is no issue with this collinearity.

Further, he stated (and this is present on page 65 of his excellent Regression Modeling Strategies book) that collinearity between variables constructed in an algebraic fashion like restricted cubic splines is not an issue as multicollinearity only matters when that collinearity changes from sample to sample.

$\endgroup$
  • $\begingroup$ You can always orthogonalize the generated splines (for example the rcsgen Stata command uses Gram-Schmidt orthogonalizaton) $\endgroup$ – boscovich Sep 25 '13 at 21:12
7
$\begingroup$

The multicollinearity can lead to numerical problems when estimating such a function. This is why some use B-splines (or variations on that theme) instead of restricted cubic splines. So, I tend to see restricted cubic splines as one potentially usefull tool in a larger toolbox.

$\endgroup$
  • $\begingroup$ Hi Maarten, when you say numerical problems do you refer to convergence or such? If the regression converged and appears to do well on a test set - do you conclude the situation is not a problem? $\endgroup$ – B_Miner Sep 25 '13 at 11:17
  • $\begingroup$ If there are numerical problems then lack of convergence is a likely (but not a necessary) consequence. Unrealistic coefficient estimates and/or unrealisticly high standard errors are other possible consequences. $\endgroup$ – Maarten Buis Sep 25 '13 at 11:37
  • $\begingroup$ Maarten that is only the case if you are using non-modern software. Modern software uses things like the QR decomposition rendering this a non-problem. $\endgroup$ – Frank Harrell Sep 25 '13 at 21:23
  • $\begingroup$ Methods like QR decomposition helped a lot. However, you can still break modern software, it has just become harder to do so. $\endgroup$ – Maarten Buis Sep 26 '13 at 10:21
  • 1
    $\begingroup$ Don't look at individiual coefficients and standard errors so much. Look at the entire fitted curve. $\endgroup$ – Frank Harrell Mar 21 '17 at 12:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.