From what I understand about the data, there should not be a Y-intercept in the model (assume that for any regression model we obtain from mlr,l1,l2, the intercept is of negligible value). I understand that MLR doesnt care about the distribution of the data - one can even fit it without standardizing anything, whereas for L1/L2, the value matters - hence the question.
For example, let's say we have 2 features X1,X2, both with values between [-1,1]:
X1 distribution: 90% of values lie greater than 0.8 ie num(x > 0.8) + num(x < -0.8) = 80% of data, with fewer points as we go further the tails
X2 distribution: 90% lie within 0.2 ie [-0.2,0.2]
Standardizing by standard deviation would cause X2 to have extreme values at the tails. Standardizing by Max of abs(X) (for this dataset means not standardizing since X1,X2 is already [-1,1]) would cause X1 to heavily penalised if in the "wrong" direction as compared to Y.
Should the solution be:
- Standardize X1 by std & Standardize X2 by abs value?
- Still standardize by std
I am concerned because incorrectly choosing the standardization procedure would like cause the features to be unfairly weighted in the Model - L1 would mean wrongly choosing features, L2 would mean incorrectly weighted.
Let me know your thoughts or correct me if Im wrong. Thanks in advance!