1
$\begingroup$

I have a dataset, consisting of 4 continuous and 1 categorical (three levels) indepentend variable. On this dataset, I want to perform a multiple linear regression with a regularization (specifically Lasso/Rdige/Elasticnet).

Let's assume I use Python with pandas and sklearn as my tools. My sample dataset in Python looks like this:

df = pd.DataFrame(
    data=[[4939.095037, 2.669234, 16.215479, 96.020074,  -0.023664, 2],
          [4386.046313, 5.043865, 40.406483, 201.266814, -0.478503, 2],
          [4827.804643, 7.605047, 23.051799, 84.555656,   2.998284, 1],
          [4277.870976, 6.447839, 37.703208, 156.311657, -0.569904, 2],
          [2187.534673, 0.961723, 27.030330, 57.628101,   1.466355, 2],
          [5978.240745, 7.402969, 73.276094, 106.040676,  3.125664, 0],
          [8684.959385, 7.930216, 31.960732, 141.064857, -0.693754, 1],
          [6533.489282, 3.633354, 34.480927, 134.808051, -4.912898, 0],
          [8374.502249, 7.897356, 40.525879, 127.356577,  2.891337, 2],
          [6488.086242, 7.520293, 27.731389, 86.830189,   0.560935, 2]],
    columns=['a', 'b', 'c', 'd', 'e', 'cat'])

Now I use dummy coding to encode the categorical variable cat with k=3 levels into k-1=2 levels. For this purpose I apply pd.get_dummies, but of couse sklearn.preprocessing.OneHotEncoder yields the same results:

df_dc = pd.get_dummies(df, columns=['cat'], drop_first=True)

Now I scale the data by subtracting the mean and sclaing to unit variance:

scaler = skl.preprocessing.StandardScaler().fit(df_dc)
df_scld = pd.DataFrame(data=scaler.transform(df_dc), columns=df_dc.columns)

The dummy encoded cat. var. is now quite "obfuscated", but still has unique values per level.

For simplicity, feeding this dataset into polynomial transformation with interaction terms (degree 2 or 3) is omitted here, but I usually make use of this (either before or after standardization -> see question 2).
Then depending on the dimensionality of the problem into a PCA and finally into the linear regression model with regularization.


My questions:

  1. Should I standardize/scale my data WITH or WITHOUT dummy coded cat. variables?

In other words: Should the dummy coded cat. vars. be scaled or not? Googling and searching CV there seem to be different opinions on this, but I can't find any "ascertained" opinions on this topic. (Some say: retaining binary 0-1 is important, other say that it doesn't hurt to scale the variables, except for human readability.)
Additional information: I'm talking mainly about standardization by subtracting the mean and scaling to unit variance. Of course min-max-scaling won't affect binary variables. :)

  1. What is the generally recommended preprocessing order in total?

    I currently either use path no. 1 or 2, My last source (see below) suggests no. 3, but I highly doubt that...

    1. Dummy coding -> polynomial transformation -> standardization/scaling -> fit model
    2. Dummy coding -> standardization/scaling -> polynomial transformation -> fit model
    3. polynomial transformation -> Dummy coding -> standardization/scaling -> fit model
  2. Is there any advantage/disadvantage of dropping the most frequent level of dummy encoded variables?

In my example this would be dropping level 2. Most algorithms simply drop the first level (here level 0), but I've read many times that dropping the most frequent level should be preferred.

  1. Is dropping a level required at all when using a regularized regression method?

General opinion seems to be yes, but reading the sklearn doc for the parameter drop, it seems like only non-regularized methods or neural networks require dropping the first level.


Some sources I've been looking up:

$\endgroup$
1
$\begingroup$

We do standardization/normalization to put our features in $[0,1]$ or $[-1,1]$ range. Let suppose we are using min-max normalization to put the values in the range $[0,1]$. The answer of your question are as follows.

  1. Should I standardize/scale my data WITH or WITHOUT dummy coded cat. variables?

    There is no clear Yes/No answer to this question. But it is not mandatory to do scaling of one-hot-encoded or dummy-encoded features. The intuition behind why it is not mandatory to do scaling is as follows.
    Let say you have got two encoded vectors as $A = [0 1 0]$ and $B = [1 0 0]$, you can see that $|A| = \sqrt{0^2+1^2+0^2}\;\;and\;\;|B|=\sqrt{1^2+0^2+0^2}$ will always be equals to $1$ and the distance between them will be $\sqrt{1^2 + 1^2} = \sqrt{2} = 1.41$. So why you should not do standardization is clear from this, as you can see the magnitude of the one-hot encoded features is $1$ and the distance between them is $\sqrt{2}$ hence the variance in this one-hot encoded feature is not that much so as to standardize them. But when you should consider to do standardization? It is when, when you have vectors like $[111011]$ and $[000001]$ in which the variability is very high

  2. What is the generally recommended preprocessing order in total?

    You should do Dummy coding -> polynomial transformation -> standardization/scaling -> fit model.
    The Reason behind doing polynomial featurization before standardization is quite simple. If you first do standardization then your variable will be in range $[0,1]$ and then squaring them will make the polynomial feature very small due to which your model will not sustain the numerical stability of this feature

Your next questions are not clear to me. Please elaborate them

Hope this helps!

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ Many thanks, this already helps alot! Concerning 2.: But how do I scale interaction terms of dummy coded variables and continuous variables? Just scale by unit variance or also subtract the mean? Because when subtracting the mean, the 0-states will get lost... What range of $\sigma$ would you consider a safe range? currently the standard deviation of my un-scaled dummy coded columns is between 0.3 and 0.7, where of course all numeric columns are sigma=1. $\endgroup$ – Scotty1- May 1 at 17:15
  • $\begingroup$ To 3.: With the question I wanted to ask, if it is relevant for the quality of the fit, which of the k dummy coded columns will be dropped to get the k-1 columns, assuming all columns have vastly different 1-value frequencies. Or is it irrelevant for the model quality? $\endgroup$ – Scotty1- May 1 at 17:15
  • $\begingroup$ As i already explained you should avoid using scaling for one-hot-encoding. But still if you want to do it then you have to follow the standard rule (x-mean)/variance and don't worry about the 0-states as now you have transformed your features which is not 0-1 vector now. To your third question, if you have linear collinearity between the features then only you should use the drop function.For more information on that refer - stats.stackexchange.com/questions/231285/… $\endgroup$ – Deepak Chaudhary May 1 at 17:40
  • $\begingroup$ Thanks again! I don't want to scale my one-hot-encoded variables, thus I was hoping for this clarification. :) $\endgroup$ – Scotty1- May 1 at 18:30

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.