54

Your approach is entirely correct. Although data transformations are often undervalued as "preprocessing", one cannot emphasize enough that transformations in order to optimize model performance can and should be treated as part of the model building process. Reasoning: A model shall be applied on unseen data which is in general not available at the time ...


52

Standardization isn't required for logistic regression. The main goal of standardizing features is to help convergence of the technique used for optimization. For example, if you use Newton-Raphson to maximize the likelihood, standardizing the features makes the convergence faster. Otherwise, you can run your logistic regression without any standardization ...


50

The third way is correct. Exactly why is covered in wonderful detail in The Elements of Statistical Learning, see the section "The Wrong and Right Way to Do Cross-validation", and also in the final chapter of Learning From Data, in the stock market example. Essentially, procedures 1 and 2 leak information about either the response, or from the future, from ...


37

For the regression model using the standardized variables, we assume the following form for the regression line $$ \mathbb E[Y] =\beta_{0}+\sum_{j=1}^{k}\beta_{j}z_{j}, $$ where $z_{j}$ is the j-th (standardized) regressor, generated from $x_j$ by subtracting the sample mean $\bar x_j$ and dividing by the sample standard deviation $S_j$: $$ z_j = \frac{x_j ...


35

According Tibshirani (THE LASSO METHOD FOR VARIABLE SELECTION IN THE COX MODEL, Statistics in Medicine, VOL. 16, 385-395 (1997)), who literally wrote the book on regularization methods, you should standardize the dummies. However, you then lose the straightforward interpretability of your coefficients. If you don't, your variables are not on an even ...


35

No. Random Forests are based on tree partitioning algorithms. As such, there's no analogue to a coefficient one obtain in general regression strategies, which would depend on the units of the independent variables. Instead, one obtain a collection of partition rules, basically a decision given a threshold, and this shouldn't change with scaling. In other ...


31

If you use logistic regression with LASSO or ridge regression (as Weka Logistic class does) you should. As Hastie,Tibshirani and Friedman points out (page 82 of the pdf or at page 63 of the book): The ridge solutions are not equivariant under scaling of the inputs, and so one normally standardizes the inputs before solving. Also this thread does.


29

1- Min-max normalization retains the original distribution of scores except for a scaling factor and transforms all the scores into a common range [0, 1]. However, this method is not robust (i.e., the method is highly sensitive to outliers. 2- Standardization (Z-score normalization) The most commonly used technique, which is calculated using the arithmetic ...


28

To illustrate the numerical stability issue mentioned by @cbeleites, here is an example from Simon Wood on how to "break" lm(). First we'll generate some simple data and fit a simple quadratic curve. set.seed(1); n <- 100 xx <- sort(runif(n)) y <- .2*(xx-.5)+(xx-.5)^2 + rnorm(n)*.1 x <- xx+100 b <- lm(y ~ x+I(x^2)) plot(x,y) lines(x, predict(...


27

Lasso regression puts constraints on the size of the coefficients associated to each variable. However, this value will depend on the magnitude of each variable. It is therefore necessary to center and reduce, or standardize, the variables. The result of centering the variables means that there is no longer an intercept. This applies equally to ridge ...


26

Ridge regression regularize the linear regression by imposing a penalty on the size of coefficients. Thus the coefficients are shrunk toward zero and toward each other. But when this happens and if the independent variables does not have the same scale, the shrinking is not fair. Two independent variables with different scales will have different ...


24

@Aymen is right, you don't need to normalize your data for logistic regression. (For more general information, it may help to read through this CV thread: When should you center your data & when should you standardize?; you might also note that your transformation is more commonly called 'normalizing', see: How to verify a distribution is normalized?) ...


21

A short recap. Given a model $y=X\beta+\varepsilon$, where $X$ is $n\times p$, $\hat\beta=(X'X)^{-1}X'y$ and $\hat y=X\hat\beta=X(X'X)^{-1}X'y=Hy$, where $H=X(X'X)^{-1}X'$ is the "hat matrix". Residuals are $$e=y-\hat y=y-Hy=(I-H)y$$ The population variance $\sigma^2$ is unknown and can be estimated by $MSE$, the mean square error. Semistudentized residuals ...


19

Sometimes standardization helps for numerical issues (not so much these days with modern numerical linear algebra routines) or for interpretation, as mentioned in the other answer. Here is one "rule" that I will use for answering the answer myself: Is the regression method you are using invariant, in that the substantive answer does not change with ...


19

The short answer: yes, you do need to worry about your data's distribution not being normal, because standardization does not transform the underlying distribution structure of the data. If $X\sim\mathcal{N}(\mu, \sigma^2)$ then you can transform this to a standard normal by standardizing: $Y:=(X-\mu)/\sigma \sim\mathcal{N}(0,1)$. However, this is possible ...


19

This stems from the property of variance. For a random variable $X$ and a constant $a$, $\mathrm{var}(aX)=a^2\mathrm{var}(x)$. Therefore, if you divide the data by its standard deviation ($\sigma$), $\mathrm{var}(X/\sigma)=\mathrm{var}(X)/\sigma^2=\sigma^2/\sigma^2=1$.


16

You have identified an important but perhaps under-appreciated issue: there is no single one-size-fits-all approach to normalize categorical variables in penalized regression. Normalization tries to ensure that penalization is applied fairly across all predictors, regardless of the scale of measurement. You don't want penalization of a predictor based on ...


15

The credit for this answer goes to @ttnphns who explained everything in the comments above. Still, I would like to provide an extended answer. To your question: Are the LDA results on standardized and non-standardized features going to be exactly the same? --- the answer is Yes. I will first give an informal argument, and then proceed with some math. ...


15

Yes, it is desirable to standartize the data while learning Gaussian processes regression. There are a number of reasons: In common Gaussian processes regression model we suppose that output $y$ has zero mean, so we should standartize $y$ to match our assumption. For many covariance function we have scale parameters in covariance functions. So, we should ...


14

For k-NN, I'd suggest normalizing the data between $0$ and $1$. k-NN uses the Euclidean distance, as its means of comparing examples. To calculate the distance between two points $x_1 = (f_1^1, f_1^2, ..., f_1^M)$ and $x_2 = (f_2^1, f_2^2, ..., f_2^M)$, where $f_1^i$ is the value of the $i$-th feature of $x_1$: $$ d(x_1, x_2) = \sqrt{(f_1^1 - f_2^1)^2 + (...


13

In general, you are right to worry about scaling the responses. If you optimize a function of the kind that LASSO is based on, $$ \min_{\beta} || Y - X\beta ||_{2}^{2} + \lambda || \beta ||_1, $$ then scaling the response $Y$ with some constant $\alpha$, $$ \min_{\beta} || \alpha Y - X\beta ||_{2}^{2} + \lambda || \beta ||_1 $$ leads to $$ \min_{\beta} \...


12

Andrew Gelman's blog post, When to standardize regression inputs and when to leave them alone, is also worth a look. This part in particular is relevant: For comparing coefficients for different predictors within a model, standardizing gets the nod. (Although I don’t standardize binary inputs. I code them as 0/1, and then I standardize all other numeric ...


12

Can I test for correlation between variables before standardize them? I am not quite sure what should I do first. Correlation will be the same regardless whether you calculate it before or after standardization. To see this, it is enough to know that correlation is invariant to scale. Take $b \in \mathbb{R}$ and $a>0$, then $$ \begin{aligned} \text{Corr}...


11

SVM tries to maximize the distance between the separating plane and the support vectors. If one feature (i.e. one dimension in this space) has very large values, it will dominate the other features when calculating the distance. If you rescale all features (e.g. to [0, 1]), they all have the same influence on the distance metric.


11

Technically, z-scoring does not depend on any distributional assumptions, such as normality. It's just a way of describing how far observations are from the mean, no matter what the distribution happens to be. So no harm in z-scoring non-normal variables. The main caveat is that z-scores tend to be more informative for distributions that are at least ...


11

I tracked down the standardization process of glmnet and documented it on the thinklab Platform there. This includes a comparison of the different ways to use standardization with glmnet. Long story short, if you let glmnet standardize the coefficients (by relying on the default standardize = TRUE), glmnet performs standardization behind the scenes and ...


11

Yes Yes You standardize variables to compare the importance of independent variables in determining the outcome variables. You may want to center a variable when you use an interaction term--its effect will be meaningfully interpretable if the minimum value of one of the interacted variables is not zero. If you are regressing different outcome variables (...


11

If you use glmnet, the scaling is performed by the package. You don't need to worry about scaling the test set because the "coefficients are always returned on the original scale". By default: glmnet(x, y, [...] standardize = TRUE, intercept = TRUE, standardize.response = FALSE [...]) As for the standardization of the response, it should not change the ...


11

What the scale function does in R is answered here. Basically, it both re-scales the mean value to be zero and the standard deviation to be 1. Several points are worth noting 1) If the original variables were not normally distributed (ND), the scaled variables will not be ND either. Conversely, if the original variables are ND, the rescaled distributions ...


Only top voted, non community-wiki answers of a minimum length are eligible