In regression, standardization is recommended in ordered to assess the relative importance of predictors. However there seems to be an assumption of normality? How would the interpretation work for predictors following different non normal distributions? It seems confusing...
2
-
3$\begingroup$ Predictors are assumed to be fixed and known. There is no assumption that they have any distribution in regression (unless you have large enough errors in estimating the predictors - which I don't think you're asking about). $\endgroup$– JimBCommented Jul 12, 2016 at 20:29
-
2$\begingroup$ @Anton The distribution of the predictors, whether normal or not, does not affect the validity of your regression model. $\endgroup$– horaceTCommented Jul 12, 2016 at 22:07
Add a comment
|
2 Answers
$\begingroup$
$\endgroup$
9
Standardization does not change your underlying distribution. It only changes the units of measurement. Also, in regression there is not assumption regarding distribution of your independent variables. The requirement is only that the residuals of your model be normally distributed.
-
$\begingroup$ Thanks for the reply, what I meant is that the assumption of normality seems to be assumed for a meaningful interpretation. An increment of one standard deviation of an exponentially distribution doesn't seem comparable to an increment of 1 standard deviation of an gaussian random variable. What I mean is that when we standardise, if the data follow a different distribution then 1 standard deviation is referring to a difference coverage of the distribution. I hope I am missing something here? $\endgroup$– AntonCommented Jul 13, 2016 at 10:08
-
$\begingroup$ Conceptually I would like to explain a model with increments as proportion of what I have observed in the population. So a "medium increment" would correspond to 68% of the population boundary for that predictor, which is one standard deviation if normal. Perhaps it makes more sense to normalize the data by using quantiles? $\endgroup$– AntonCommented Jul 13, 2016 at 10:15
-
$\begingroup$ I'm not sure you are expressing your question meaningfully. When you build a linear regression, the slope for example simply says how much will y change, when x changes. For example, knowing that temperature influences pressure by a slope of 2 is the relationship between the two variables. Knowing that temperatures greater than 25 degrees occur only 2% of the time does not change the relationship to pressure. So the interpretation of importance of a variable and its distribution is in effect two different things. $\endgroup$ Commented Jul 13, 2016 at 10:33
-
$\begingroup$ regarding your example I would interpret the model as "an increase of 1 in temperature will produce a change of 2 in pressure" if the data is standardized I would interpret the same model as "an increase of 1 standard deviation in temperature will produce a change of 2 in pressure". However I would like to express "a small/medium/large increase in temperature will result in change dy in y". When multiple predictors are present I would like small/medium/large to have the same meaning across predictors. $\endgroup$– AntonCommented Jul 13, 2016 at 10:38
-
$\begingroup$ That still wouldn't depend on your distribution! A small change (1 degree = 2 units in pressure), a large change (10 degree = 20 in pressure). Since the regression has quantified the relationship for you, why would you want to obfuscate it behind small medium large. You could simple state the relationship as is. $\endgroup$ Commented Jul 13, 2016 at 10:42
$\begingroup$
$\endgroup$
2
However there seems to be an assumption of normality?
As pointed out by @Arun Jose there's no assumption of normality regarding the independent variables. You can read about the Gauss-Markov Assumptions
How would the interpretation work for predictors following different non normal distributions?
This is a very popular question: check out this answer. We can break it down in two parts: centering and scaling. While the answer to this question concerning scaling is trivial, centering is more interesting.
Centering will result in predictors with mean zero.
[Centering] makes it so the intercept term is interpreted as the expected value of YiYi when the predictor values are set to their means. Otherwise, the intercept is interpreted as the expected value of YiYi when the predictors are set to 0, which may not be a realistic or interpretable situation (e.g. what if the predictors were height and weight?) see here]
Centering plays a role in two scenarios:
The only case I can think of off the top of my head where centering is helpful is before creating power terms. Lets say you have a variable, XX, that ranges from 1 to 2, but you suspect a curvilinear relationship with the response variable, and so you want to create an X2X2 term. If you don't center XX first, your squared term will be highly correlated with XX, which could muddy the estimation of the beta. Centering first addresses this issue [source]
and
An analogous case that I forgot to mention is creating interaction terms. If an interaction / product term is created from two variables that are not centered on 0, some amount of collinearity will be induced (with the exact amount depending on various factors). Centering first addresses this potential problem same source again and this
-
$\begingroup$ Thanks for the reply and links, what I meant is assumptions for a meanignful interpretation. I would like to create increments that reflect "small", "medium" , "large" and I thought that the standard deviation would provide this. However if I define "medium" as "what is experienced by 50% of the population or less" than it clearly depends on the underlying distribution of the data. $\endgroup$– AntonCommented Jul 13, 2016 at 10:21
-
1$\begingroup$ @Anton To bin data in such manner, e.g. 'small', 'medium, 'large', I would directly estimate the density from observations and try to normalize the resulting distribution. $\endgroup$– MikeCommented Oct 15, 2019 at 18:21