1
$\begingroup$

I have a logistic regression model with 5 continuous independent variables and a categorical variable.

I scaled down one of the continuous variable using the formula Scaled_Value = NewMin_value + Ratio between new and old Range * (Value - Old_Min _Value)

I checked the shape of scaled version of the variable using skewness and kurtosis. They were same as the original variable.

Next, I fitted logistic reg model again with all the variables - 4 continuous + 1 categorical + scaled version of continuous variable instead of the original variable.

Compared to the first model, I see that scores and coefficients for all variables are same except for the intercept and coefficient of scaled variable which have now changed. Rest are all same.

I'm trying to understand the reason behind this? How the scores remained unchanged. And why intercept and coefficient of scaled variable has to change. THanks.

$\endgroup$
1
  • $\begingroup$ Can anyone help me understand this? $\endgroup$ Oct 14, 2014 at 7:10

1 Answer 1

3
$\begingroup$

Your scaled variable, denote it $Z_1$, is an affine function of the original variable,

$$Z_1 = c_0 + c_1X_1$$ where $c_0,c_1$ are constants and determined by the equation you describe in your question.

In logistic regression, we are estimating a conditional probability for which we have assumed a specific functional form

$$P(Y\mid X_1,..X_k) = \frac 1{1+e^{-g(\mathbf X)}}, \;\;g(\mathbf X) = b_0+b_1X_1+...+b_kX_k $$

You inserted $Z_1$ instead of $X_1$. So in this case you essentially specified

$$g(\mathbf X_{-1}, Z_1) = b_0+b_1(c_0 + c_1X_1)+...+b_kX_k $$ $$= (b_0+b_1c_0) + b_1c_1X_1+...+b_kX_k = d_0 + d_1X_1+...+b_kX_k$$

In other words you are back to the original regressor matrix, but with a different intercept, and a different coefficient for the scaled variable. I hope this helps.

$\endgroup$
4
  • $\begingroup$ Excellent, it helps a lot! So if score remains the same, it means that (b0 + b1c0 + b1c1X1) = (b0 + b1X1). That is, b1X1 = b1c1X1 + b1c0. Right? $\endgroup$ Oct 14, 2014 at 15:49
  • $\begingroup$ What does your model tells you? You can check whether this is verified. $\endgroup$ Oct 14, 2014 at 15:52
  • $\begingroup$ I don't think, that you are correct @Alecos. Why do you say, that we are 'back to the original regressor matrix'? The true reason why scaled variables have no effect on predictions is that during gradient update step the error governs the optimization process. And the final value of weights is determined by this error and not the scale of inputs. However, by scaling the input nonuniformly one can make the life of SGD arbitrarily bad :-0. Albeit in the end the result stays the same $\endgroup$ Mar 24, 2017 at 10:27
  • $\begingroup$ @VastAcademician Thanks for the feedback. The mathematics of my answer are correct, and specifically target the question of the OP as to why the OP obtained different coefficient estimates (the OP did not ask about predictions). The "back to the same regressor matrix" is an "as if" comment, to show what effects an affine scalling of one variable has here. $\endgroup$ Mar 24, 2017 at 12:47

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.