I would like to know your insights / opinions, about what is good approach:

MAIN QUESTION: Remove highly correlated columns BEFORE or AFTER converting nominals to dummy variables?

The difference in final dataset can be quite huge, because:

  • converting to dummies can create many columns (often having many same numbers / zeros), that can be strongly correlated and removed.
  • on other, removing these columns can skew / break the logical concept of converting nominals to dummies

Imagine common situation in life of data-scientist

  1. we are doing predictive regression like Kaggle House Prices
  2. prepare basic steps for data cleaning & preprocessing
    1. setting correct column types
    2. imputing missing values
    3. removing ID columns
    4. convert nominals + ordinals to numbers
    5. optionally remove outliers
    6. optionally fix skewed distributions to normal
    7. optionally remove strongly correlated columns

The goal is making most accurate predictive model = to train model with smallest error on test set.

Removing correlated variables can help in many models - especially in linear regression models

I google'd a lot and I can give myself one simple answer:

  • Let's try both (decorrelate before / after dummies) and compare results

This is completely practical and valid answer, but it is quite general and applicable almost everywhere :)

So I am looking for more insights / experiences / opinions of skilled people.

In most data-science tutorials online, the steps are in this order:

  1. decorrelation
  2. conversion to dummies - is executed after decorrelation

I am curious:

  1. why it is so common to see doing it in this specific order (decorrelation first, to-dummies next)?
  2. why it is not done in opposite order? - because conversion to dummies could bring many new correlated columns into the dataset again?
  • 5
    $\begingroup$ What is the goal of your analysis, and why does it necessitate removing correlated variables? $\endgroup$ – Matthew Drury Oct 27 '17 at 20:19
  • $\begingroup$ Need much more to opine, but happy to do so. $\endgroup$ – eSurfsnake Oct 28 '17 at 6:15
  • 2
    $\begingroup$ A strange question. If you are removing some of dummies this means you are - depending on the type of your subsequent analysis - either deleting categories from the nominal variable or merging the "deleted" categories in one. So, why the decision what to do with separate categories of nominal variable X should depend on some variable Y? I'm not to say it's illegal, only that it must have a special reason for. $\endgroup$ – ttnphns Oct 28 '17 at 10:35
  • $\begingroup$ Yes, I have same feeling it is a strange question :) Maybe it is good to ask: In case - conversion to dummy variables creates correlated columns - should we remove them ? What if correlation is so strong, that it creates almost same or strongly correlated columns - is one of them redundant? $\endgroup$ – Stefan Simik Oct 28 '17 at 18:28
  • $\begingroup$ "Removing correlated variables can help in many models - especially in linear regression models" Not if you're interested in prediction. I'm not sure where you read this is true, but it is false. $\endgroup$ – Matthew Drury Oct 29 '17 at 1:03

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.