# Tag Info

30

In regression, it is often recommended to center the variables so that the predictors have mean $0$. This makes it so the intercept term is interpreted as the expected value of $Y_i$ when the predictor values are set to their means. Otherwise, the intercept is interpreted as the expected value of $Y_i$ when the predictors are set to 0, which may not be a ...

23

You have come across a common belief. However, in general, you do not need to center or standardize your data for multiple regression. Different explanatory variables are almost always on different scales (i.e., measured in different units). This is not a problem; the betas are estimated such that they convert the units of each explanatory variable into ...

21

In addition to the remarks in the other answers, I'd like to point out that the scale and location of the explanatory variables does not affect the validity of the regression model in any way. Consider the model $y=\beta_0+\beta_1x_1+\beta_2x_2+\ldots+\epsilon$. The least squares estimators of $\beta_1, \beta_2,\ldots$ are not affected by shifting. The ...

13

In the business world, "normalization" typically means that the range of values are "normalized to be from 0.0 to 1.0". "Standardization" typically means that the range of values are "standardized" to measure how many standard deviations the value is from its mean. However, not everyone would agree with that. It's best to explain your definitions ...

13

Although terminology is a contentious topic, I prefer to call "explanatory" variables, "predictor" variables. When to standardise the predictors: A lot of software for performing multiple linear regression will provide standardised coefficients which are equivalent to unstandardised coefficients where you manually standardise predictors and the response ...

9

Standardization is all about the weights of different variables for the model. If you do the standardisation "only" for the sake of numerical stability, there may be transformations that yield very similar numerical properties but different physical meaning that could be much more appropriate for the interpretation. The same is true for centering, which is ...

8

Let the variable be $x$, its mean be $\mu$, and its standard deviation $\sigma$, so that the standardized variable is $z = (x-\mu)/\sigma$. By expanding $z^2$ and collecting like powers of $x$ you can rewrite your model as \eqalign{ y &= \beta_0 + \beta_1 x + \beta_2 z^2 + \varepsilon \\ &= \beta_0 + \beta_1 x + \beta_2 ... 6 This is to make the variance of the statistic 1. The statistic, S, is a linear combination of independent z-scores: S = \frac{\sum_{i=1}^{k} Z_{i}}{\sqrt{k}} $$so the variance of S is$$ {\rm var}(S) = {\rm var}\left( \frac{\sum_{i=1}^{k} Z_{i}}{\sqrt{k}} \right) = \frac{1}{k} \sum_{i=1}^{k} {\rm var}( Z_{i}) = \frac{1}{k} \cdot k = 1$$If ... 6 (You may start from the after line section, for a shorter answer) To begin with, you are absolutely right saying that it firstly depends on the purposes of your analysis: forecasting of average price (at macro level) or a particular price (at micro level), causal analysis of consumer preferences (district, size, age, number of bedrooms, gas, travelling to ... 6 I prefer "solid reasons" for both centering and standardization (they exist very often). In general, they have more to do with the data set and the problem than with the data analysis method. Very often, I prefer to center (i.e. shift the origin of the data) to other points that are physically/chemically/biologically/... more meaningful than the mean (see ... 5 Here is a possible plyr solution. Note that it relies on the base transform() function. my.df <- data.frame(x=rnorm(100, mean=10), sex=sample(c("M","F"), 100, rep=T), group=gl(5, 20, labels=LETTERS[1:5])) library(plyr) ddply(my.df, c("sex", "group"), transform, x.std = scale(x)) (We can check whether it works ... 5 1. Which component tests to combine You need to determine whether it is meaningful to combine your battery of ability tests to form an overall composite. This is separate to the issue of the weightings you use for the component variables. This links in to both general literature on validity and scale construction, as well as more domain-specific literature ... 5 I can't see that standardization is a good idea in ordinary regression or with a longitudinal model. It makes predictions harder to obtain and doesn't solve a problem that needs solving, usually. And what if you have x and x^2 in the model. How do you standardize x^2? What if you have a continuous variable and a binary variable in the model? How ... 5 Rescaling the input features is just a linear transformation. There's no right or wrong way of rescaling outside a problem context. If you want to map the range 1 - 100 to the range 1 - 10 linearly you should do:$$ x \leftarrow \frac{x - 1}{99} \times 9 + 1  This maps 1 to 1 and 100 to 10 and it will make the durations have the same range as the other ...

5

First off, standardization usually is taken to be subtraction of the mean division by the standard deviation. The result has a mean 0 and standard deviation of 1. Dividing by the variance will be wrong for any variable that is not a pure number. One of the reasons for standardization is to remove any influence of the units of measurement. The ...

5

rstandard() produces standardised residuals via normalisation to unit variance using the overall error variance of the residuals/model. rstudent() produces Studentized residuals in the same way, but it uses a leave-one-out estimate of the error variance. The key line in rstandard() is res <- infl$wt.res/(sd * sqrt(1 - infl$hat)) where sd is defined ...

5

The notation '$X_k \in N(\mu,\sigma)$' in your source means that the random variable $X_k$ has a normal distribution with mean $\mu$ and standard deviation $\sigma$. The second symbol in $N(\cdot,\cdot)$ denotes the standard deviation, NOT the variance. This was brought to my attention by @Dilip Sarwate (see comment below). It is more common to see the ...

4

Normalization rescales the values from to a range of [0,1]. This might useful in some cases where all parameters need to have the same positive scale, but outliers from data set are lost. Xchanged = (X - Xmin)/(Xmax-Xmin) Standardization rescales data to have a mean of 0 and standard deviation of 1 (unit variance). Xchanged = x-mean/sd For most ...

4

Of course you can normalize your parameters, this would also increase the speed of the learning algorithm. In order to have comparable $\beta$ at the end of the execution of the algorithm you should, for each feature $x_i$, compute its mean $\mu_i$ and its range $r_i = \max_i - \min_i$. Then you change each $r[x_i]$ value, ie the value of feature $x_i$ for ...

4

In general I don't recommend scaling or standardization unless it's absolutely necessary. The advantage or appeal of such a process is that, when an explanatory variable has a totally different physical dimension and magnitude from the response variable, scaling through division by standard deviation may help in terms of numerical stability, and enables one ...

4

In case you use gradient descent to fit your model, standardizing covariates may speed up convergence (because when you have unscaled covariates, the corresponding parameters may inappropriately dominate the gradient). To illustrate this, some R code: > objective <- function(par){ par[1]^2+par[2]^2} #quadratic function in two variables with a minimum ...

4

Unless your algorithms have huge differences in performance and you have huge numbers of test cases, you won't be able to detect differences by just looking at the performance. However, you can make use of apaired design: run all three algorithms on exactly the same train/test split of a data set, and do not aggregate the test results into % correct, but ...

4

In linear regression it is not necessary to mean center or normalize either your x or y variable. Some people think it improves interpretation; I tend to disagree and prefer the raw units. But both are statistically fine. The main reason I don't like scaling is that it makes the parameter estimates refer to standard deviations derived from the data set you ...

4

Cohen's d is a measure of effect size. Standard deviation is a measure of spread. The standard deviation is used in calculating Cohen's d, but other than that there is very little relation - they measure different things, they require different input and they answer different questions.

3

Simply scale your explanatory variables to having mean of zero and variance of one before you put them in the model. Then the coefficients will all be comparable. The mixed effects nature of the model doesn't impact on this issue. The best way to do it, and least likely to go wrong, is to use scale() before you fit the model.

3

@Simone makes some good points, so I will just throw in a couple of complementary tidbits. Although normalization can help with things like speed, logistic regression does not make assumptions about the distributions of your predictor variables. Thus, you don't have to normalize. Second, while adding a squared term can lead to overfitting (and you need to ...

3

For the calculation of basic epidemiologic estimates in a pretty neat, intuitive way that's considerably easier than R but surprisingly powerful, you cannot beat EpiSheet (link is an Excel file) by Ken Rothman. It's a simple Excel file and yet does an amazing amount. It will, for example, in the "Risk Data" tab, give you the 90, 95 and 99% Confidence ...

3

A few quick points about logs The following R code is a reminder that the log of a negative number is not a number and that the log of zero is negative infinity. Thus, if you are going to take a log of a z-score, you first need to make all values obtained greater than zero. > values <- c(-2, -1, 0, .001, .1, 1, 10) > data.frame(values=values, ...

3

I am not entirely sure that's relevant, but this might help (at least as a food thought): Peter Bentler worked a lot on comparing unit-weighted scales (sums of scores/symptoms) as compared to more complicated factor-analysis-based scores. Something like this: http://www.google.com/search?q=peter+bentler+Unit-weighted+Composites. With a tiny sample size, ...

3

The answer is simple, but you're not going to like it: it depends. If you value 1 standard deviation from both scores equally, then standardization is the way to go (note: in fact, you're studentizing, because you're dividing by an estimate of the SD of the population). If not, it is likely that standardization will be a good first step, after which you can ...

Only top voted, non community-wiki answers of a minimum length are eligible