I have a dataset, consisting of 4 continuous and 1 categorical (three levels) indepentend variable. On this dataset, I want to perform a multiple linear regression with a regularization (specifically Lasso/Rdige/Elasticnet).
Let's assume I use Python with
sklearn as my tools. My sample dataset in Python looks like this:
df = pd.DataFrame( data=[[4939.095037, 2.669234, 16.215479, 96.020074, -0.023664, 2], [4386.046313, 5.043865, 40.406483, 201.266814, -0.478503, 2], [4827.804643, 7.605047, 23.051799, 84.555656, 2.998284, 1], [4277.870976, 6.447839, 37.703208, 156.311657, -0.569904, 2], [2187.534673, 0.961723, 27.030330, 57.628101, 1.466355, 2], [5978.240745, 7.402969, 73.276094, 106.040676, 3.125664, 0], [8684.959385, 7.930216, 31.960732, 141.064857, -0.693754, 1], [6533.489282, 3.633354, 34.480927, 134.808051, -4.912898, 0], [8374.502249, 7.897356, 40.525879, 127.356577, 2.891337, 2], [6488.086242, 7.520293, 27.731389, 86.830189, 0.560935, 2]], columns=['a', 'b', 'c', 'd', 'e', 'cat'])
Now I use dummy coding to encode the categorical variable
k=3 levels into
k-1=2 levels. For this purpose I apply
pd.get_dummies, but of couse
sklearn.preprocessing.OneHotEncoder yields the same results:
df_dc = pd.get_dummies(df, columns=['cat'], drop_first=True)
Now I scale the data by subtracting the mean and sclaing to unit variance:
scaler = skl.preprocessing.StandardScaler().fit(df_dc) df_scld = pd.DataFrame(data=scaler.transform(df_dc), columns=df_dc.columns)
The dummy encoded cat. var. is now quite "obfuscated", but still has unique values per level.
For simplicity, feeding this dataset into polynomial transformation with interaction terms (degree 2 or 3) is omitted here, but I usually make use of this (either before or after standardization -> see question 2).
Then depending on the dimensionality of the problem into a PCA and finally into the linear regression model with regularization.
- Should I standardize/scale my data WITH or WITHOUT dummy coded cat. variables?
In other words: Should the dummy coded cat. vars. be scaled or not? Googling and searching CV there seem to be different opinions on this, but I can't find any "ascertained" opinions on this topic. (Some say: retaining binary 0-1 is important, other say that it doesn't hurt to scale the variables, except for human readability.)
Additional information: I'm talking mainly about standardization by subtracting the mean and scaling to unit variance. Of course min-max-scaling won't affect binary variables. :)
What is the generally recommended preprocessing order in total?
I currently either use path no. 1 or 2, My last source (see below) suggests no. 3, but I highly doubt that...
- Dummy coding -> polynomial transformation -> standardization/scaling -> fit model
- Dummy coding -> standardization/scaling -> polynomial transformation -> fit model
- polynomial transformation -> Dummy coding -> standardization/scaling -> fit model
Is there any advantage/disadvantage of dropping the most frequent level of dummy encoded variables?
In my example this would be dropping level 2. Most algorithms simply drop the first level (here level 0), but I've read many times that dropping the most frequent level should be preferred.
- Is dropping a level required at all when using a regularized regression method?
General opinion seems to be yes, but reading the sklearn doc for the parameter
drop, it seems like only non-regularized methods or neural networks require dropping the first level.
Some sources I've been looking up:
- CV: centering and scaling dummy variables
- CV: Significance of categorical predictor in logistic regression
- towards data science: Preprocessing with sklearn: a complete and comprehensive guide. Honestly: I don't trust "towards data science". I've read so many false statetments and explanations there, that my first reaction towards articles on this site is mistrust...