I would like to know your insights / opinions, about what is good approach:
MAIN QUESTION: Remove highly correlated columns BEFORE or AFTER converting nominals to dummy variables?
The difference in final dataset can be quite huge, because:
- converting to dummies can create many columns (often having many same numbers / zeros), that can be strongly correlated and removed.
- on other, removing these columns can skew / break the logical concept of converting nominals to dummies
Imagine common situation in life of data-scientist
- we are doing predictive regression like Kaggle House Prices
- prepare basic steps for data cleaning & preprocessing
- setting correct column types
- imputing missing values
- removing ID columns
- convert nominals + ordinals to numbers
- optionally remove outliers
- optionally fix skewed distributions to normal
- optionally remove strongly correlated columns
The goal is making most accurate predictive model = to train model with smallest error on test set.
Removing correlated variables can help in many models - especially in linear regression models
I google'd a lot and I can give myself one simple answer:
- Let's try both (decorrelate before / after dummies) and compare results
This is completely practical and valid answer, but it is quite general and applicable almost everywhere :)
So I am looking for more insights / experiences / opinions of skilled people.
In most data-science tutorials online, the steps are in this order:
- conversion to dummies - is executed after decorrelation
I am curious:
- why it is so common to see doing it in this specific order (decorrelation first, to-dummies next)?
- why it is not done in opposite order? - because conversion to dummies could bring many new correlated columns into the dataset again?