Hot answers tagged standardization
24
In regression, it is often recommended to center the variables so that the predictors have mean $0$. This makes it so the intercept term is interpreted as the expected value of $Y_i$ when the predictor values are set to their means. Otherwise, the intercept is interpreted as the expected value of $Y_i$ when the predictors are set to 0, which may not be a ...
18
You have come across a common belief. However, in general, you do not need to center or standardize your data for multiple regression. Different explanatory variables are almost always on different scales (i.e., measured in different units). This is not a problem; the betas are estimated such that they convert the units of each explanatory variable into ...
17
In addition to the remarks in the other answers, I'd like to point out that the scale and location of the explanatory variables does not affect the validity of the regression model in any way.
Consider the model $y=\beta_0+\beta_1x_1+\beta_2x_2+\ldots+\epsilon$.
The least squares estimators of $\beta_1, \beta_2,\ldots$ are not affected by shifting. The ...
12
Although terminology is a contentious topic, I prefer to call "explanatory" variables, "predictor" variables.
When to standardise the predictors:
A lot of software for performing multiple linear regression will provide standardised coefficients which are equivalent to unstandardised coefficients where you manually standardise predictors and the response ...
11
In the business world, "normalization" typically means that the range of values are "normalized to be from 0.0 to 1.0". "Standardization" typically means that the range of values are "standardized" to measure how many standard deviations the value is from its mean. However, not everyone would agree with that. It's best to explain your definitions ...
7
Let the variable be $x$, its mean be $\mu$, and its standard deviation $\sigma$, so that the standardized variable is $z = (x-\mu)/\sigma$. By expanding $z^2$ and collecting like powers of $x$ you can rewrite your model as
$$\eqalign{
y &= \beta_0 + \beta_1 x + \beta_2 z^2 + \varepsilon \\
&= \beta_0 + \beta_1 x + \beta_2 ...
6
This is to make the variance of the statistic 1. The statistic, $S$, is a linear combination of independent $z$-scores:
$$ S = \frac{\sum_{i=1}^{k} Z_{i}}{\sqrt{k}} $$
so the variance of $S$ is
$$ {\rm var}(S) = {\rm var}\left( \frac{\sum_{i=1}^{k} Z_{i}}{\sqrt{k}} \right) = \frac{1}{k} \sum_{i=1}^{k} {\rm var}( Z_{i}) = \frac{1}{k} \cdot k = 1$$
If ...
6
(You may start from the after line section, for a shorter answer) To begin with, you are absolutely right saying that it firstly depends on the purposes of your analysis: forecasting of average price (at macro level) or a particular price (at micro level), causal analysis of consumer preferences (district, size, age, number of bedrooms, gas, travelling to ...
6
I prefer "solid reasons" for both centering and standardization (they exist very often). In general, they have more to do with the data set and the problem than with the data analysis method.
Very often, I prefer to center (i.e. shift the origin of the data) to other points that are physically/chemically/biologically/... more meaningful than the mean (see ...
5
I can't see that standardization is a good idea in ordinary regression or with a longitudinal model. It makes predictions harder to obtain and doesn't solve a problem that needs solving, usually. And what if you have $x$ and $x^2$ in the model. How do you standardize $x^2$? What if you have a continuous variable and a binary variable in the model? How ...
5
1. Which component tests to combine
You need to determine whether it is meaningful to combine your battery of ability tests to form an overall composite. This is separate to the issue of the weightings you use for the component variables.
This links in to both general literature on validity and scale construction, as well as more domain-specific literature ...
4
Here is a possible plyr solution. Note that it relies on the base transform() function.
my.df <- data.frame(x=rnorm(100, mean=10),
sex=sample(c("M","F"), 100, rep=T),
group=gl(5, 20, labels=LETTERS[1:5]))
library(plyr)
ddply(my.df, c("sex", "group"), transform, x.std = scale(x))
(We can check whether it works ...
4
Of course you can normalize your parameters, this would also increase the speed of the learning algorithm.
In order to have comparable $\beta$ at the end of the execution of the algorithm you should, for each feature $x_i$, compute its mean $\mu_i$ and its range $r_i = \max_i - \min_i$. Then you change each $r[x_i]$ value, ie the value of feature $x_i$ for ...
4
In case you use gradient descent to fit your model, standardizing covariates may speed up convergence (because when you have unscaled covariates, the corresponding parameters may inappropriately dominate the gradient). To illustrate this, some R code:
> objective <- function(par){ par[1]^2+par[2]^2} #quadratic function in two variables with a minimum ...
4
Rescaling the input features is just a linear transformation. There's no right or wrong way of rescaling outside a problem context. If you want to map the range 1 - 100 to the range 1 - 10 linearly you should do:
$$
x \leftarrow \frac{x - 1}{99} \times 9 + 1
$$
This maps 1 to 1 and 100 to 10 and it will make the durations have the same range as the other ...
3
For the calculation of basic epidemiologic estimates in a pretty neat, intuitive way that's considerably easier than R but surprisingly powerful, you cannot beat EpiSheet (link is an Excel file) by Ken Rothman. It's a simple Excel file and yet does an amazing amount.
It will, for example, in the "Risk Data" tab, give you the 90, 95 and 99% Confidence ...
3
Standardization is all about the weights of different variables for the model.
If you do the standardisation "only" for the sake of numerical stability, there may be transformations that yield very similar numerical properties but different physical meaning that could be much more appropriate for the interpretation. The same is true for centering, which is ...
3
In general I don't recommend scaling or standardization unless it's absolutely necessary. The advantage or appeal of such a process is that, when an explanatory variable has a totally different physical dimension and magnitude from the response variable, scaling through division by standard deviation may help in terms of numerical stability, and enables one ...
3
@Simone makes some good points, so I will just throw in a couple of complementary tidbits. Although normalization can help with things like speed, logistic regression does not make assumptions about the distributions of your predictor variables. Thus, you don't have to normalize. Second, while adding a squared term can lead to overfitting (and you need to ...
3
The answer is simple, but you're not going to like it: it depends. If you value 1 standard deviation from both scores equally, then standardization is the way to go (note: in fact, you're studentizing, because you're dividing by an estimate of the SD of the population).
If not, it is likely that standardization will be a good first step, after which you can ...
3
A few quick points about logs
The following R code is a reminder that the log of a negative number is not a number and that the log of zero is negative infinity. Thus, if you are going to take a log of a z-score, you first need to make all values obtained greater than zero.
> values <- c(-2, -1, 0, .001, .1, 1, 10)
> data.frame(values=values, ...
3
I am not entirely sure that's relevant, but this might help (at least as a food thought): Peter Bentler worked a lot on comparing unit-weighted scales (sums of scores/symptoms) as compared to more complicated factor-analysis-based scores. Something like this: http://www.google.com/search?q=peter+bentler+Unit-weighted+Composites. With a tiny sample size, ...
3
Unless your algorithms have huge differences in performance and you have huge numbers of test cases, you won't be able to detect differences by just looking at the performance.
However, you can make use of apaired design:
run all three algorithms on exactly the same train/test split of a data set, and
do not aggregate the test results into % correct, but ...
2
I'd just say: be careful with that. Standarization is needed only when some variable(s) dominates the dissimilarity score just because it is expressed in "smaller units"; let's say that you have a variable that is truly equal for all elements, but there is some, very small variability due to the measurement error. Now if you'd normalize this value, you'll ...
2
Spearman's correlation is just Pearson's correlation using ranks (see the Wikipedia page), so any transformation of the data that preserves their ordering (and so gives the same ranks) will give precisely the same value for Spearman's correlation.
Pearson's correlation doen't have that property, but it is the case that it is unaffected by any linear ...
2
Simply scale your explanatory variables to having mean of zero and variance of one before you put them in the model. Then the coefficients will all be comparable. The mixed effects nature of the model doesn't impact on this issue.
The best way to do it, and least likely to go wrong, is to use scale() before you fit the model.
2
You are correct about zeroing the means of the columns of $A$ and $b$.
However, as for adjusting the norms of the columns of $A$, consider what would happen if you started out with a normed $A$, and all the elements of $x$ were of roughly the same magnitude. Then let us multiply one column by, say, $10^{-6}$. The corresponding element of $x$ would, in an ...
2
In theory, the scale of your inputs are irrelevant to logistic regression. You can "theoretically" multiply $X_1$ by $10^{10^{10^{10}}}$ and the estimate for $\beta_1$ will adjust accordingly. It will be $10^{-10^{10^{10}}}$ times smaller than the original $\beta_1$, due to the invariance property of MLEs.
But try getting R to do the above adjusted ...
2
I don't know how representative my experience is, but I've met several people with somewhat strong opinions on centering. (Which is weird, really, because who cares?) It may be that you have run into this yourself. Often, centering is discussed in the context of multicollinearity. In general, I don't think centering really does much; if $x_1$ & $x_2$ ...
2
In the Blinder Oaxaca decomposition, which is an econometric technique normally used to compare logged income, the interest is not just in the difference in the means, but also whether the difference is more due to:
returns on human capital (so the coefficient values differ between the male and female equations), or
human capital levels and other ...
Only top voted, non community-wiki answers of a minimum length are eligible
