# Avoiding perfect multicollinearity in machine learning/regression

Suppose that I'm trying to use regression predict the GDP of a country.

Assume I'm using the following features (variables):

• Dummy variables indicating which continent a country is from (hence there are dummy variables for Asia, Africa, etc).

• Dummy variables indicating whether a country's population is less than 50 million, between 50 to 100 million, or more than 100 million

• Two-way interactions of the above dummies (i.e. continent dummies interacted with population dummies)

If I don't have an intercept (constant), do I need to omit one dummy variable for either population or continent to be the base category, so as to avoid perfect multicollinearity?

Do I need to omit some of the interactions so as to avoid perfect multicollinearity?

Do the answers to the above questions depend on whether I am using machine learning techniques like Lasso, or just doing "standard" statistical regressions without regularization/shrinkage?

If you want to measure the effect of some variable on GDP, then you would need to avoid multicollinearity (perfect or otherwise). For example, to measure the effect of population size on GDP, if you have something like number of social security numbers, you might want to remove one or the other because they should be almost perfectly correlated.

Suppose you didn't. Let the number of social security numbers = population size = $x$. Suppose the true effect $\beta_1$ is 1.

Then the following choices of $\hat{\beta_0}, \hat{\beta_1}, \hat{\beta_2}$ might be possible solutions for respectively the intercept, population, and number of social security numbers

$\hat{\beta_0} + 1x +0x$

$\hat{\beta_0} + 0x +1x$

$\hat{\beta_0} + 0.5x +0.5x$

etc...

So if someone asked you the effect ${\beta_1}$ of population on GDP, it would be hard to say--it appears to be anything between 0 and 1 depending on the coefficient given to the number of social security numbers. Besides your estimation of effect, many other things break down. For example, your estimate of the variance of your estimate of the effect is no longer valid.

However, on the other hand, if someone asked you to give a prediction $\hat{y}$ of GDP based on the number of social security numbers and population, you could just give them any of these

$\hat{y_1} = \hat{\beta_0} + 1x +0x$

$\hat{y_2} = \hat{\beta_0} + 0x +1x$

$\hat{y_3} =\hat{\beta_0} + 0.5x +0.5x$

= etc...

Because your predictions $\hat{y_1} =\hat{y_2} =\hat{y_3}$

In the second case, you're not really positing that there exists $\beta_1$ or trying to find it. You're constructing an object $\{\hat{\beta_0}, \hat{\beta_1}, \hat{\beta_2}\}$ based on data that is some random realization of many possible solutions. This is the typical machine learning workflow and the deliverable is generally a product--some technology that can be used to predict (a sklearn model is literally an object).

The statistical approach also gives you this object, but this is not the objective generally. Rather the statistical approach seeks to uncover some kind of truth about $\beta_1$--the regression in this case is a procedure (SAS proc reg) rather than an object. The real output of this procedure is an estimate of $\beta_1$, which is an increase in knowledge. Clearly if you have a very good estimate of $\beta_1$, and your original model is correct, then you can make predictions, so you have the same thing that you got with the ML approach, but you also can make statements about $\beta_1$, unlike in the ML approach. This is reductionist; there are many researchers in machine learning who are more interested in population parameters and researchers in statistics who are more interested in creating predictive systems.

For interaction terms, the same is true: estimating effect is not the same as making a prediction.

It does not matter whether you use lasso or ridge.

Note that if your estimation of the variance of the effect is not valid, then your estimation of the variance of the prediction is not valid. So one cannot directly calculate prediction intervals in the second case. You can, however, construct empirical prediction intervals. These however depend on your data, since you create these intervals by sampling. But I suppose the intervals from the equations also depend on your data...