MLR - Eliminating multicollinearity when predictors are transformations of others

Question

I am applying a multiple linear regression on a data set, where some of the predictors are "transformations" of others (however, I'm not entirely sure if they are linear transformations or not).

For the sake of an example, let's say that we have three predictors, $A$, $B$, and $C$ that completely explain the variance of some dependent variable $Y$. However, $B$ and $C$ are highly correlated since $C$ is a transformation of $B$.

The transformation is $C_i = \sum_{j=0}^{23} V_j B_{i-j}$ where V is a vector of 24 numbers.

My questions for you are the following:

1. Is it possible to completely eliminate multicollinearity among the predictors seeing that I know exactly how some are transformations of others?

2. If so, how do I go about eliminating this multicollinearity?

Thank you!

Edit: thanks to @curiositasisasinbutstillcuriou, I am getting very close to the solution of my question; however, I need confirmation that my Python code to retrieve the original coefficients of my predictors makes sense; the majority of the following code was taken from https://www.statology.org/principal-components-regression-in-python/ while the last line is my attempt at retrieving the original coefficients.

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
from sklearn.preprocessing import scale 
from sklearn import model_selection
from sklearn.model_selection import RepeatedKFold
from sklearn.model_selection import train_test_split
from sklearn.decomposition import PCA
from sklearn.linear_model import LinearRegression
from sklearn.metrics import mean_squared_error

### Fit the PCR Model ###

# X is a dataframe of the original predictors
# y is a dataframe of the dependent variable

# scale predictor variables
pca = PCA()
X_reduced = pca.fit_transform(scale(X))

# define cross validation method
cv = RepeatedKFold(n_splits=10, n_repeats=3, random_state=1)

regr = LinearRegression()
mse = []

# calculate MSE with only the intercept
score = -1*model_selection.cross_val_score(regr,
           np.ones((len(X_reduced),1)), y, cv=cv,
           scoring='neg_mean_squared_error').mean()    
mse.append(score)

# calculate MSE using cross-validation, adding one component at a time
for i in np.arange(1, len(X.columns)):
    score = -1*model_selection.cross_val_score(regr,
               X_reduced[:,:i], y, cv=cv, scoring='neg_mean_squared_error').mean()
    mse.append(score)
    
# plot cross-validation results    
plt.plot(mse)
plt.xlabel('Number of Principal Components')
plt.ylabel('MSE')
plt.title('hp')

# determine n_components based on the lowest MSE

### Use the Final Model to Make Predictions ###

# split the dataset into training (70%) and testing (30%) sets
X_train,X_test,y_train,y_test = train_test_split(X,y,test_size=0.3,random_state=0) 

# scale the training and testing data
X_reduced_train = pca.fit_transform(scale(X_train))[:,:n_components]
X_reduced_test = pca.transform(scale(X_test))[:,:n_components]

# train PCR model on training data 
regr = LinearRegression()
regr.fit(X_reduced_train, y_train)

# calculate RMSE
pred = regr.predict(X_reduced_test)
np.sqrt(mean_squared_error(y_test, pred))

# find coefficients for original predictors
orig_coef = np.matmul(regr.coef_, pca.components_[:n_components,:])

Answer 1

Per Frank Harrell's comment (see below) this kind of multicollinearity (produced by your transformations), will not be a problem because it will be consistent in sample and out of sample.

All the same, if you wanted to rule out multicollinearity affecting your regression, here are two straightforward options:

1. You can standardize your predictors and then apply PCA. Then your predictors will no longer be multicollinear, although your model may not be better from a predictive standpoint. You can convert the coefficients for the PCA variables to the original variables by extracting the PCA rotations and doing matrix multiplication.

2. You can also do regression using a tree-based model instead. The performance of a tree-based model should not be strongly impacted by multicollinearity. But, as far as I know, you will not be able to obtain regression coefficients; you will need to look at an alternative measure--like variable importance. See more here: https://medium.com/@manepriyanka48/multicollinearity-in-tree-based-models-b971292db140