I was using Hastie's 2005 Elastic Net to fit a linear regression model with corrected penalization using a 12MM x 769 observations. I experimented in both R and Python. I was fitting the models using the cross validation method. My question is why the default scaling method on each software is different? glmnet does standardization, while elastic net does normalization. The coefficients that each model select are different on each software. Which method is known to do a better selection?
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
1
Scaling and normalization generally help optimizers with convergence and help humans with interpreting regression coefficients. Neither one is best in all cases, but some might perform better on specific datasets than others. Normalizing, like a MinMaxScaler might ‘squish’ all values to between [0,1]. If there are a lot of extreme values in the dataset, rounding error after scaling might cause loss of information. Conversely, it might be the extreme values that are of interest in which case one might want them to be scaled into the same range as the rest of your variables and could benefit from a MinMaxScaler. There is an interesting blog post where several scaling methods were compared, but none were found to be categorically better. One could follow these methods as an example to determine which is best for their use case.
https://towardsdatascience.com/normalization-vs-standardization-quantitative-analysis-a91e8a79cebf
-
4$\begingroup$ It is more to do with the coherence of regularising the explanatory variables. If two variables are on different scales then regularisation will affect them differently. Sure it might help the optimisation too but that secondary. Standard linear regression works fine most of the time with no regularisation. $\endgroup$ Jul 31, 2019 at 7:27