Multicollinearity: does if matter which variable I remove? Say I'm running a multiple linear regression. I have 4 explanatory variables: A, B, C and D.
Pearsons correlation coefficient for A and B is 0.90. I decide to remove either A or B prior to running the multiple linear regression.
Will A and B both have exactly the same correlation coefficient with C and D, in which it doesn't matter which one of A or B I remove?
Or could A and B both have different correlation coefficients with C and D, in which I should remove the least correlated explanatory variable, A or B, prior to running the multiple linear regression?
 A: *

*
Q : Will A and B both have exactly the same correlation coefficient with C
  and D, in which it doesn't matter which one of A or B I remove?

A and B don't necessarily have the same correlation coefficient with C and D. Think of $X=[A,B,C]\sim N_3(0,\Sigma)$ where $\Sigma= \left( \begin{array}{ccc}
1 & 0.9 & 0.1 \\
0.9 & 1 & 0.3 \\
0.1 & 0.3 & 1 
\end{array} \right)$.
For sample version, check it through experiment.
library(mvtnorm)
set.seed(123)
Sigma <- c(1, 0.9, 0.1, 0.9, 1, 0.3, 0.1, 0.3, 1)
Sigma <- matrix(Sigma, nrow = 3)
X <- rmvnorm(n = 100, mean = rep(0,3), sigma = Sigma)
cor(X)


*
Q : could A and B both have different correlation coefficients with C and
  D, in which I should remove the least correlated explanatory variable,
  A or B

This is not a good question. Suppose cor(A,C)=0.40, cor(A,D)=0.67, cor(B,C)=0.56, and cor(B,D)=0.27. B is more correlated to C than A, but A is more correlated to D than B. How do you define "the least correlated explanatory variable"? Should it be A or B? This situation is possible. Please try the following R code.
set.seed(123)
data <- matrix(rnorm(40), nrow=10)
data[,3] <- -data[,3]
colnames(data) <- c("A","B","C","D")
cor(data)


*If removing one variable between A and B is the only option, then the solutions I can think of are 


*

*drop the variable which is less correlated with the response

*drop the variable which has larger Variance Inflation Factor (VIF)

*fit 2 new models : one removing A and one removing B, and to see which new model suffers less multicollinearity. To measure multicollinearity, you can look at the VIF of each present variable or the Condition Number of $X'X$ where $X$ is the present design matrix

*use variable selection method like stepwise regression and lasso regression to (hopefully) remove one variable


I am not sure which are valid solutions or which is the best one, so any comment is appreciated.
