1
$\begingroup$

I have two datasets that have exactly the same 1701 predictors, but one has 936 subjects and the other has 547 subjects. (The initial rationale for creating these two different datasets was to see whether the conclusions would depend on the sample size and the percentage of missing data.) I then put both datasets through the same machine learning pipeline that comprises the following steps:

  • Scaling
  • Imputation of missing data with K-nearest neighbours
  • Dropping of predictors with zero variance
  • Dropping of predictors with a variance inflation factor (VIF) >= 5.0
  • Encoding of ordinal and categorical predictors
  • Feature selection using recursive feature elimination
  • Machine learning model

I then noticed that the 4th step - the dropping of collinear predictors using VIF - resulted in the number of predictors dropping from 1641 to 954 for the larger dataset (with 936 subjects), but the number of predictors didn't budge at all (1604 to 1604) for the smaller dataset (with 547 subjects). For the latter, I verified that the intermediate dataset entered into the VIF algorithm indeed led to all calculated VIFs being < 5.0, hence the retention of all predictors wasn't due to an error.

That brings me to my question: Does having a larger sample size (more subjects) somehow result in an increase in multi-collinearity amongst a dataset's predictors, after missing data imputation with K-nearest neighbours (or any other imputation method)?

I tried looking for answers on this forum as well as via a Google search, but couldn't find anything relevant. I'd be really grateful for any advice on this matter. Thank you very much!

=======================

Addendum made after Ian Barnett's answer:

I've experimented using the code kindly written by Ian Barnett, and it seems that increasing sample size doesn't appear to affect the degree of post-imputation collinearity in that particular example, but increasing the percentage of missing data does. (Hope I'm doing this correctly - I noticed that increasing pmiss led to an unexpected increase in the frequencies (height of the histogram) that made them appear 'saturated', as shown in the images).

enter image description here

enter image description here

enter image description here

enter image description here

$\endgroup$

1 Answer 1

1
$\begingroup$

Imputation can indeed increase multi-collinearity. I've demonstrated this with a simple bivariate normal example below. In this example I generate 100 bivariate normal random variables with correlation $\rho=.4$. I then randomly create some missingness in one of the variables ($X_2$) leaving the other variable ($X_1$) complete with no missingness. After this I created a simple imputation regression model that uses only the complete observations to predict $X_2$ from $X_1$. I stored the correlation of $X_1$ and $X_2$ both pre and post imputation and repeated this 1000 times to generate the histograms below. You will notice that the complete data has correlations hovering around the true $\rho$, while the imputed data has higher correlations in general.

This is likely what is happening in your case. Increasing sample size does not increase multi-collinearity, but imputation certainly can as I've demonstrated here.



library(mvtnorm)

set.seed(124)
B=1000
rho=.4
n=100
pmiss=.3

impucor=rep(NA,B)
origcor=rep(NA,B)
for(b in 1:B){
  # generate bivariate data
  dat=rmvnorm(n,mean=c(0,0),sigma=matrix(c(1,rho,rho,1),nrow=2))
  datmis=dat
  
  # create missing data in the second variable, X2, while leaving X1 compelte
  IDall=sample(1:n)
  IDmiss=IDall[1:floor(n*pmiss)]
  IDcomplete=IDall[(floor(n*pmiss)+1):n]
  datmis[IDmiss,2]=NA
  
  # imputation model
  X1c=datmis[IDcomplete,1]
  X2c=datmis[IDcomplete,2]
  lm.out =lm(X2c~X1c)
  betacoef=coef(lm.out)
  
  # create imputed data matrix by using predictions from imputation regression model
  datimp = datmis
  for(i in 1:length(IDmiss)){
    datimp[IDmiss[i],2]=betacoef[1]+betacoef[2]*datimp[IDmiss[i],1]
  }
  # store the correlation between X1 and X2 in the original data as
  # well as in the imputed data after missingness is induced
  impucor[b]=cor(datimp)[1,2]
  origcor[b]=cor(dat)[1,2]
}



plt1=hist(origcor,breaks=30)
plt2=hist(impucor,breaks=30)
plot(plt1,col=rgb(0,0,1,1/4),xlim=c(0,1),main="Correlation of imputed versus original data",xlab="Correlation")
plot(plt2,col=rgb(1,0,0,1/4),add=T)
colors=c(rgb(1,0,0,1/4),rgb(0,0,1,1/4))
legend("topright",c("Imputed","Original"),col=colors,pch=15)

enter image description here

$\endgroup$
5
  • $\begingroup$ imputation is informed by the non-missing data, so one would expect a higher level of correlation unless random imputation is used. You could impute using random values in the range of existing values. Imagine that for each row of missing data you did this a few times, you could change the balance of the data some, but you would also be adding uncorrelated values in those missing variable locations. $\endgroup$ Nov 18, 2021 at 21:52
  • $\begingroup$ Thank you very much! Your example demonstrates very nicely how multi-collinearity is increased after imputation. Not being familiar with R, I've naively adjusted n and pmiss in your code (I only made changes to those 2 parameters; I left the rest of the code untouched) to examine whether an increase in sample size and/or missingness would increase the degree of post-imputation collinearity. I've edited my question with these new figures. It appears that increasing pmiss increases the degree of post-imputation collinearity, but increasing n does not. Does this seem correct to you? $\endgroup$ Nov 19, 2021 at 1:06
  • $\begingroup$ @Schrodinger Yes, that does make sense. Indeed simply increasing sample size (n) does not impact multi-collinearity, but increasing missingness would increase multi-collinearity. $\endgroup$ Nov 19, 2021 at 1:24
  • $\begingroup$ @IanBarnett Thank you very much for your advice on this! On a related note, do you think that this result - i.e. increasing missingness leads to higher multi-collinearity after imputation - can be mitigated by the choice of imputation strategy? Intuitively, I would think that all imputation strategies suffer from this drawback, but is there a strategy that would be less afflicted by this? (MICE perhaps?) From my basic understanding, imputation methods inherently exploit multi-collinearity to impute missing data, and would thus always strengthen existing multi-collinearity. $\endgroup$ Nov 19, 2021 at 17:30
  • $\begingroup$ @Schrodinger Yes the imputation choice should be able to impact this effect of increasing multicollinearity. In the example I used in the code, one way of improving this could be by adding noise to the prediction rather than just using the mean imputation based on the regression model. This is more reminiscent of a parametric bootstrap. In other words, in the code I wrote above I imputed the missing value of $X_2$ with $\beta_0 + \beta_1 X_1$, but perhaps using $\beta_0 + \beta_1 X_1 + \epsilon$ for some $\epsilon$ coming from a noise distribution derived form the data might help. $\endgroup$ Nov 20, 2021 at 21:36

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.