6
$\begingroup$

There is a standard result for linear regression that the regression coefficients are given by

$$\mathbf{\beta}=(\mathbf{X^T X})^{-1}\mathbf{X^T y}$$

or

$(\mathbf{X^T X})\mathbf{\beta}=\mathbf{X^T y} \tag{2}\label{eq2}$

Scaling the explanatory variables does not affect the predictions. I have tried to show this algebraically as follows.

The response is related to the explanatory variables via the matrix equation $\mathbf{y}=\mathbf{X \beta} \tag{3}\label{eq3}$

$\mathbf{X}$ is an $n \times (p+1)$ matrix of n observations on p explanatory variables. The first column of $\mathbf{X}$ is a column of ones.

Scaling the explanatory variables with a $(p+1) \times (p+1) $ diagonal matrix $\mathbf{D}$, whose entries are the scaling factors $ \mathbf{X^s} = \mathbf{XD} \tag{4}\label{eq4}$

$\mathbf{X^s}$ and $\mathbf{\beta^s}$ satisfy $\eqref{eq2}$:

$$(\mathbf{D^TX^T XD})\mathbf{\beta^s} =\mathbf{D^TX^T y}$$

so

$$\mathbf{X^T XD}\mathbf{\beta^s} =\mathbf{X^T y}$$

$$\Rightarrow \mathbf{D \beta^s} = (\mathbf{X^T X)^{-1}}\mathbf{X^T y}=\mathbf{\beta}$$

$\Rightarrow \mathbf{\beta^s}=\mathbf{D}^{-1}\mathbf{\beta} \tag{5}\label{eq5}$

This means if an explanatory variable is scaled by $d_i$ then the regression coefficient $\beta_i$is scaled by $1/d_i$ and the effect of the scaling cancels out, i.e. considering predictions based on scaled values, and using $\eqref{eq4},\eqref{eq5},\eqref{eq3}$

$$\mathbf{y^s}=\mathbf{X^s \beta^s} = \mathbf{X D D^{-1}\beta}=\mathbf{X \beta}=\mathbf{y}$$ as expected.

Now to the question.

For logistic regression without any regularization, it is suggested, by doing regressions with and without scaling the same effect is seen


fit <- glm(vs ~ mpg, data=mtcars,family=binomial)

print(fit)

Coefficients:
(Intercept)          mpg  
    -8.8331       0.4304  
mtcars$mpg <- mtcars$mpg * 10

fit <- glm(vs ~ mpg, data=mtcars,family=binomial)

print(fit)

Coefficients:
(Intercept)          mpg  
   -8.83307      0.04304  

When the variable mpg is scaled up by 10, the corresponding coefficient is scaled down by 10.

  1. How could this scaling property be proved (or disproved ) algebraically for logistic regression?

I found a similar question relating to the effect on AUC when regularization is used.

  1. Is there any point to scaling explanatory variables in logistic regression, in the absence of regularization?

Thanks.

$\endgroup$
9
$\begingroup$

Here is a heuristic idea:

The likelihood for a logistic regression model is $$ \ell(\beta|y) \propto \prod_i\left(\frac{\exp(x_i'\beta)}{1+\exp(x_i'\beta)}\right)^{y_i}\left(\frac{1}{1+\exp(x_i'\beta)}\right)^{1-y_i} $$ and the MLE is the arg max of that likelihood. When you scale a regressor, you also need to accordingly scale the coefficients to achieve the original maximal likelihood.

$\endgroup$
  • $\begingroup$ That's useful +1 thank-you. Anything to add regarding the second point about the reasons for doing (or not doing) the scaling? $\endgroup$ – PM. Mar 25 at 15:36
  • 2
    $\begingroup$ Not sure. Given that in practice, MLEs are computed via numerical (e.g. Newton-Raphson) algorithms, they might be more susceptible to stability issues when regressors live on extremely different scales. Also, of course, different users may prefer different units of measurement (say, km vs miles). $\endgroup$ – Christoph Hanck Mar 25 at 15:40
6
$\begingroup$

Christoph has a great answer (+1). Just writing this because I can't comment there.

The crucial point here is that the likelihood only depends on the coefficients $\beta$ through the linear term $X \beta$. This makes the likelihood unable to distinguish between "$X \beta$" and $(XD) (D^{-1}\beta)$", causing the invariance you've noticed.

To be specific about this, we need to introduce some notation (which we can do since we're writing an answer!). Let $y_i | x_i \stackrel{ind.}{\sim} \mathrm{bernoulli}\left[ \mathrm{logit}^{-1} (x_i^T \beta) \right]$ be independent draws according to the logistic regression model, where $x_i \in \mathbb{R}^{p+1}$ is the measured covariates. Write the likelihood of the $i^{th}$ observation as $l(y_i, x_i^T \beta)$.

To introduce the change of coordinates, write $\bar{x}_i = D x_i$, where $D$ is diagonal matrix with all diagonal entries nonzero. By definition of maximum likelihood estimation, we know that maximum likelihood estimators $\hat{\beta}$ of the data $\{y_i | x_i\}$ satisfy that $$\sum_{i=1}^n l(y_i, x_i^T \beta) \leq \sum_{i=1}^n l(y_i, x_i^T \hat\beta) \tag{1}$$ for all coefficients $\beta \in \mathbb{R}^p$, and that maximum likelihood estimators for the data $\{y_i | \bar{x}_i\}$ satisfy that $$\sum_{i=1}^n l(y_i, \bar{x}_i^T \alpha) \leq \sum_{i=1}^n l(y_i, \bar{x}_i^T \hat\alpha) \tag{2}$$ for all coefficients $\alpha \in \mathbb{R}^p$.

In your argument, you used a closed form of the maximum likelihood estimator to derive the result. It turns out, though, (as Cristoph suggested above), all you need to do is work with the likelihood. Let $\hat{\beta}$ be a maximum likelihood estimator of the data $\{y_i | x_i\}$. Now, writing $\beta = D \alpha$, we can use equation (1) to show that $$\sum_{i=1}^n l(y_i, \bar{x}_i^T \alpha) = \sum_{i=1}^n l\left(y_i, (x_i^T D) (D^{-1} \beta)\right) \leq \sum_{i=1}^n l(y_i, x_i^T \hat\beta) = \sum_{i=1}^n l(y_i, \bar{x}_i^T D^{-1} \hat{\beta}).$$ That is, $D^{-1} \hat{\beta}$ satisfies equation (2) and is therefore a maximum likelihood estimator with respect to the data $\{y_i | \bar{x}_i\}$. This is the invariance property you noticed.

(For what it's worth, there's a lot of room for generalizing this argument beyond logistic regression: did we need independent observations? did we need the matrix $D$ to be diagonal? did we need a binary response? did we need the use logit? What notation would you change for this argument to work in different scenarios?)

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.