11
$\begingroup$

I don't have a formal definition of scale equivariance, but here's what Introduction to Statistical Learning says about this on p. 217:

The standard least squares coefficients... are scale equivariant: multiplying $X_j$ by a constant $c$ simply leads to a scaling of the least squares coefficient estimates by a factor of $1/c$.

For simplicity, let's assume the general linear model $\mathbf{y} = \mathbf{X}\boldsymbol\beta + \boldsymbol\epsilon$, where $\mathbf{y} \in \mathbb{R}^N$, $\mathbf{X}$ is a $N \times (p+1)$ matrix (where $p+1 < N$) with all entries in $\mathbb{R}$, $\boldsymbol\beta \in \mathbb{R}^{p+1}$, and $\boldsymbol\epsilon$ is a $N$-dimensional vector of real-valued random variables with $\mathbb{E}[\boldsymbol\epsilon] = \mathbf{0}_{N \times 1}$.

From OLS estimation, we know that if $\mathbf{X}$ has full (column) rank, $$\hat{\boldsymbol\beta}_{\mathbf{X}} = (\mathbf{X}^{T}\mathbf{X})^{-1}\mathbf{X}^{T}\mathbf{y}\text{.}$$ Suppose we multiplied a column of $\mathbf{X}$, say $\mathbf{x}_k$ for some $k \in \{1, 2, \dots, p+1\}$, by a constant $c \neq 0$. This would be equivalent to the matrix \begin{equation} \mathbf{X}\underbrace{\begin{bmatrix} 1 & \\ & 1 \\ & & \ddots \\ & & & 1 \\ & & & & c\\ & & & & & 1 \\ & & & & & &\ddots \\ & & & & & & & 1 \end{bmatrix}}_{\mathbf{S}} = \begin{bmatrix} \mathbf{x}_1 & \mathbf{x}_2 & \cdots & c\mathbf{x}_{k} & \cdots & \mathbf{x}_{p+1}\end{bmatrix} \equiv \tilde{\mathbf{X}} \end{equation} where all other entries of the matrix $\mathbf{S}$ above are $0$, and $c$ is in the $k$th entry of the diagonal of $\mathbf{S}$. Then, $\tilde{\mathbf X}$ has full (column) rank as well, and the resulting OLS estimator using $\tilde{\mathbf X}$ as the new design matrix is $$\hat{\boldsymbol\beta}_{\tilde{\mathbf{X}}} = \left(\tilde{\mathbf{X}}^{T}\tilde{\mathbf{X}}\right)^{-1}\tilde{\mathbf{X}}^{T}\mathbf{y}\text{.}$$ After some work, one can show that $$\tilde{\mathbf{X}}^{T}\tilde{\mathbf{X}} = \begin{bmatrix} \mathbf{x}_1^{T}\mathbf{x}_1 & \mathbf{x}_1^{T}\mathbf{x}_2 & \cdots & c\mathbf{x}_1^{T}\mathbf{x}_k & \cdots & \mathbf{x}_1^{T}\mathbf{x}_{p+1} \\ \mathbf{x}_2^{T}\mathbf{x}_1 & \mathbf{x}_2^{T}\mathbf{x}_2 & \cdots & c\mathbf{x}_2^{T}\mathbf{x}_k & \cdots & \mathbf{x}_2^{T}\mathbf{x}_{p+1} \\ \vdots & \vdots & \ddots & \vdots & \cdots & \vdots \\ c\mathbf{x}_k^{T}\mathbf{x}_1 & c\mathbf{x}_k^{T}\mathbf{x}_2 & \cdots & c^2\mathbf{x}_k^{T}\mathbf{x}_k & \cdots & c\mathbf{x}_k^{T}\mathbf{x}_{p+1} \\ \vdots & \vdots & \vdots & \vdots & \cdots & \vdots \\ \mathbf{x}_{p+1}^{T}\mathbf{x}_1 & \mathbf{x}_{p+1}^{T}\mathbf{x}_2 & \cdots & c\mathbf{x}_{p+1}^{T}\mathbf{x}_{p+1} & \cdots & \mathbf{x}_{p+1}^{T}\mathbf{x}_{p+1} \\ \end{bmatrix}$$ and $$\tilde{\mathbf{X}}^{T}\mathbf{y} = \begin{bmatrix} \mathbf{x}_1^{T}\mathbf{y} \\ \mathbf{x}_2^{T}\mathbf{y} \\ \vdots \\ c\mathbf{x}_k^{T}\mathbf{y} \\ \vdots \\ \mathbf{x}_{p+1}^{T}\mathbf{y} \end{bmatrix}$$ How do I go from here to show the claim quoted above (i.e., that $\hat{\boldsymbol\beta}_{\tilde{\mathbf{X}}} = \dfrac{1}{c}\hat{\boldsymbol\beta}_{\mathbf{X}}$)? It's not clear to me how to compute $(\tilde{\mathbf{X}}^{T}\tilde{\mathbf{X}})^{-1}$.

$\endgroup$
  • $\begingroup$ I think your $\tilde{\mathbf{X}}^{T}\tilde{\mathbf{X}}$ is not right, it's missing a $c$ multiplier in a whole row. $\endgroup$ – Firebug Nov 1 '17 at 13:04
  • 1
    $\begingroup$ Also, bear in mind the claim is $\hat{\beta}_{k, \text{new}}={1\over c}\hat{\beta}_{k, \text{old}}$, not every $\beta$. $\endgroup$ – Firebug Nov 1 '17 at 13:08
  • $\begingroup$ @Firebug Yep, I just figured that out. I'm posting an answer. $\endgroup$ – Clarinetist Nov 1 '17 at 13:08
  • 2
    $\begingroup$ You can replace all this algebra by a much simpler units analysis, because multiplying $X_j$ by $c$ merely changes its unit of measurement, and therefore the corresponding change in the units associated with its coefficient $\beta_j$ is to divide it by $c$. That doesn't prove $\hat \beta_j$ must be divided by $c$, unfortunately. However, this chain of thought might remind us that multiple regression can be carried out by a succession of regressions against one regressor at a time, where it is clear that $\hat\beta_j$ is divided by $c$, and so the proof is complete. $\endgroup$ – whuber Nov 1 '17 at 13:45
  • $\begingroup$ @whuber, while the intuition for the result is clear, it seems there simply must be a bit of algebra in furnishing a proof. After all, the scaling factor $c$ needs to be inverted. $\endgroup$ – user795305 Nov 1 '17 at 14:00
11
$\begingroup$

Since the assertion in the quotation is a collection of statements about rescaling the columns of $X$, you might as well prove them all at once. Indeed, it takes no more work to prove a generalization of the assertion:

When $X$ is right-multiplied by an invertible matrix $A$, then the new coefficient estimate $\hat\beta_A$ is equal to $\hat \beta$ left-multiplied by $A^{-1}$.

The only algebraic facts you need are the (easily proven, well-known ones) that $(AB)^\prime=B^\prime A^\prime$ for any matrices $AB$ and $(AB)^{-1}=B^{-1}A^{-1}$ for invertible matrices $A$ and $B$. (A subtler version of the latter is needed when working with generalized inverses: for invertible $A$ and $B$ and any $X$, $(AXB)^{-} = B^{-1}X^{-}A^{-1}$.)


Proof by algebra: $$\hat\beta_A = ((XA)^\prime ((XA))^{-}(XA)^\prime y = A^{-1}(X^\prime X)^{-} (A^\prime)^{-1}A^\prime y = A^{-1}\hat \beta,$$

QED. (In order for this proof to be fully general, the $^-$ superscript refers to a generalized inverse.)


Proof by geometry:

Given bases $E_p$ and $E_n$ of $\mathbb{R}^n$ and $\mathbb{R}^p$, respectively, $X$ represents a linear transformation from $\mathbb{R}^p$ to $\mathbb{R}^n$. Right-multiplication of $X$ by $A$ can be considered as leaving this transformation fixed but changing $E_p$ to $AE_p$ (that is, to the columns of $A$). Under that change of basis, the representation of any vector $\hat\beta\in\mathbb{R}^p$ must change via left-multiplication by $A^{-1}$, QED.

(This proof works, unmodified, even when $X^\prime X$ is not invertible.)


The quotation specifically refers to the case of diagonal matrices $A$ with $A_{ii}=1$ for $i\ne j$ and $A_{jj}=c$.


Connection with least squares

The objective here is to use first principles to obtain the result, with the principle being that of least squares: estimating coefficients that minimize the sum of squares of residuals.

Again, proving a (huge) generalization proves no more difficult and is rather revealing. Suppose $$\phi:V^p\to W^n$$ is any map (linear or not) of real vector spaces and suppose $Q$ is any real-valued function on $W^n$. Let $U\subset V^p$ be the (possibly empty) set of points $v$ for which $Q(\phi(v))$ is minimized.

Result: $U$, which is determined solely by $Q$ and $\phi$, does not depend on any choice of basis $E_p$ used to represent vectors in $V^p$.

Proof: QED.

There's nothing to prove!

Application of the result: Let $F$ be a positive semidefinite quadratic form on $\mathbb{R}^n$, let $y\in\mathbb{R}^n$, and suppose $\phi$ is a linear map represented by $X$ when bases of $V^p=\mathbb{R}^p$ and $W^n=\mathbb{R}^n$ are chosen. Define $Q(x)=F(y,x)$. Choose a basis of $\mathbb{R}^p$ and suppose $\hat\beta$ is the representation of some $v\in U$ in that basis. This is least squares: $x=X\hat\beta$ minimizes the squared distance $F(y,x)$. Because $X$ is a linear map, changing the basis of $\mathbb{R}^p$ corresponds to right-multiplying $X$ by some invertible matrix $A$. That will left-multiply $\hat\beta$ by $A^{-1}$, QED.

$\endgroup$
6
$\begingroup$

Define the least squares estimator $\hat\beta = \arg\min_{\beta\in\mathbb{R}^p} \|y - X \beta\|_2^2$, where the design matrix $X \in \mathbb{R}^{n \times p}$ is full rank. Assuming that the scaling matrix $S \in \mathbb{R}^{p \times p}$ is invertible.

Define this new scaled estimator $\tilde\alpha = \arg\min_{\alpha\in\mathbb{R}^p} \|y - X S \alpha\|_2^2$. This means that $$\|y - X S \tilde\alpha\|_2^2 < \|y - X S \alpha\|_2^2$$ for all $\alpha\ne \tilde\alpha$. Defining $\tilde\beta = S \tilde\alpha$, we can rewrite this displayed inequality above as $$\|y - X \tilde\beta \|_2^2 < \|y - X \beta \|_2^2$$ for all $\beta \ne \tilde\beta$. Therefore $\tilde\beta = \arg\min_{\beta \in \mathbb{R}^p} \|y - X \beta\|_2^2$, and it follows that the least squares estimator \begin{align*} \hat\beta = \tilde\beta = S \tilde\alpha. \end{align*} Due to the invertibility of the scaling matrix $S$, it follows that $\tilde\alpha = S^{-1} \hat\beta$. In our case, this only differs from $\hat\beta$ by the $k^\mathrm{th}$ entry being scaled by $\frac{1}{c}$.

$\endgroup$
  • 1
    $\begingroup$ I'm not familiar as I should be with working with $\text{arg min}$ and similar functions - could you explain the transition from your second to third line of equations? $\endgroup$ – Clarinetist Nov 1 '17 at 13:31
  • $\begingroup$ I've written it a bit differently, which should make the steps more clear. $\endgroup$ – user795305 Nov 1 '17 at 13:50
  • $\begingroup$ This is really clever. (+1) $\endgroup$ – Clarinetist Nov 1 '17 at 18:41
4
$\begingroup$

I figured this out after posting the question. If my work is correct, however, I misinterpreted the claim. The $\dfrac{1}{c}$ scaling only occurs on the one component of $\boldsymbol\beta$ corresponding to the column of $\mathbf{X}$ being multiplied by $c$.

Notice that $\mathbf{S}$, in the notation above, is a diagonal, symmetric $(p+1) \times (p+1)$ matrix and has inverse (because it is diagonal) $$\mathbf{S}^{-1} = \begin{bmatrix} 1 & \\ & 1 \\ & & \ddots \\ & & & 1 \\ & & & & \frac{1}{c}\\ & & & & & 1 \\ & & & & & &\ddots \\ & & & & & & & 1 \end{bmatrix}\text{.}$$ Note that $(\tilde{\mathbf{X}}^{T}\tilde{\mathbf{X}})^{-1}$ is a $(p+1)\times(p+1)$ matrix. Let's suppose that $$(\mathbf{X}^{T}\mathbf{X})^{-1} = \begin{bmatrix} \mathbf{z}_1 & \mathbf{z}_2 & \cdots & \mathbf{z}_k & \cdots & \mathbf{z}_{p+1} \end{bmatrix}\text{.}$$ Then it follows that $$(\tilde{\mathbf{X}}^{T}\tilde{\mathbf{X}})^{-1} = [(\mathbf{X}\mathbf{S})^{T}\mathbf{X}\mathbf{S}]^{-1} = (\mathbf{S}^{T}\mathbf{X}^{T}\mathbf{X}\mathbf{S})^{-1} = (\mathbf{S}\mathbf{X}^{T}\mathbf{X}\mathbf{S})^{-1}=\mathbf{S}^{-1}(\mathbf{X}^{T}\mathbf{X})^{-1}\mathbf{S}^{-1}\text{.}$$ Hence, $$\mathbf{S}^{-1}(\mathbf{X}^{T}\mathbf{X})^{-1} = \begin{bmatrix} \mathbf{z}_1 \\ & \mathbf{z}_2 \\ & & \ddots \\ & & & \frac{1}{c}\mathbf{z}_k \\ & & & & \ddots \\ & & & & & \mathbf{z}_{p+1} \end{bmatrix}$$ and multiplying this by $\mathbf{S}^{-1}$ has a similar effect to what multiplying $\mathbf{X}$ by $\mathbf{S}$ did - it remains the same, except $\frac{1}{c}\mathbf{z}_k$ is multiplied by $\frac{1}{c}$: $$\mathbf{S}^{-1}(\mathbf{X}^{T}\mathbf{X})^{-1}\mathbf{S}^{-1} = \begin{bmatrix} \mathbf{z}_1 \\ & \mathbf{z}_2 \\ & & \ddots \\ & & & \frac{1}{c^2}\mathbf{z}_k \\ & & & & \ddots \\ & & & & & \mathbf{z}_{p+1} \end{bmatrix}\text{.}$$ Therefore, $$\begin{align} \hat{\boldsymbol\beta}_{\tilde{\mathbf{X}}}&=\mathbf{S}^{-1}(\mathbf{X}^{T}\mathbf{X})^{-1}\mathbf{S}^{-1}(\mathbf{X}\mathbf{S})^{T}\mathbf{y} \\ &= \begin{bmatrix} \mathbf{z}_1 \\ & \mathbf{z}_2 \\ & & \ddots \\ & & & \frac{1}{c^2}\mathbf{z}_k \\ & & & & \ddots \\ & & & & & \mathbf{z}_{p+1} \end{bmatrix}\begin{bmatrix} \mathbf{x}_1^{T}\mathbf{y} \\ \mathbf{x}_2^{T}\mathbf{y} \\ \vdots \\ c\mathbf{x}_k^{T}\mathbf{y} \\ \vdots \\ \mathbf{x}_{p+1}^{T}\mathbf{y} \end{bmatrix} \\ &= \begin{bmatrix} \mathbf{z}_1\mathbf{x}_1^{T}\mathbf{y} \\ \mathbf{z}_2\mathbf{x}_2^{T}\mathbf{y} \\ \vdots \\ \frac{1}{c}\mathbf{z}_k\mathbf{x}_k^{T}\mathbf{y} \\ \vdots \\ \mathbf{z}_{p+1}\mathbf{x}_{p+1}^{T}\mathbf{y} \end{bmatrix} \end{align}$$ as desired.

$\endgroup$
  • $\begingroup$ There is a typo in $\mathbf{S}^{-1}( \mathbf{X}^{T}\mathbf{X})^{-1} \mathbf{S}^{-1}(\mathbf{X} \mathbf{S}) \mathbf{y}$. You need to transpose $(\mathbf{X}\mathbf{S})$. $\endgroup$ – JohnK Nov 1 '17 at 13:55
3
$\begingroup$

The Most Trivial Proof Ever

You start with your linear equation: $$Y=X\beta+\varepsilon$$ Now you want to change the scale of your regressors, maybe convert from the metric system to Imperial, you know kilograms to pounds, meters to yards etc. So, you come up with the conversion matrix $S=diag(s_1,s_1,\dots,s_n)$ where each $s_i$ is the conversion coefficient for variable (column) $i$ in design matrix $X$.

Let's re-write the equation: $$Y=(XS)(S^{-1}\beta)+\varepsilon$$

Now it's abundantly clear that scaling is the property of linearity of your equation, not the OLS method of estimation of coefficients. Regardless of the estimation method with linear equation you have it that when the regressors are scaled as $XS$ your new coefficients should be scaled as $S^{-1}\beta$

Proof by Algebra for OLS Only

The scaling is this: $$Z=X*diag(s_1,s_2,...,s_n)$$ where $s_i$ scale factor of each variable (column), and $Z$ a scaled version of $X$. Let's call the diagonal scale matrix $S\equiv diag(s_1,s_2,...,s_n)$. Your OLS estimator is $$\hat\beta=(X^TX)^{-1}X^TY$$ Let's plug the scaled matrix $Z$ instead of $X$ and use some matrix algebra: $$(Z^TZ)^{-1}Z^TY=(S^TX^TXS)^{-1}S^TX^TY=S^{-1}(X^TX)^{-1}S^{-1}SX^TY\\ =S^{-1}(X^TX)^{-1}X^TY=S^{-1}\hat\beta$$ So, you see how the new coefficient is simply the old coefficient scaled down, as expected.

$\endgroup$
  • 2
    $\begingroup$ I like your approaches, but am unconvinced by "the most trivial proof ever." You have implicitly assumed, and still need to show, that the rewritten model must have the same fit as the original.To put it more rigorously: if we view a fitting procedure as a function $\delta:M\to\mathbb{R}^p$, where $M$ is the set of all possible data (which we could write as the ordered pair $(X,Y)$) and $\mathbb{R}^p$ is the set of all possible coefficient estimates, then you need to demonstrate that $\delta(X,Y)=S^{-1}\delta(XS,Y)$ for all invertible $S$, all $X$, and all $Y$. (This isn't always true!) $\endgroup$ – whuber Nov 1 '17 at 19:55
  • $\begingroup$ @whuber, actually it's the other way: the reasonable fitting procedure should satisfy this condition, otherwise a simple change of unit of measure will produce a different forecast/estimate. i'll update my answer, will think about it a little $\endgroup$ – Aksakal Nov 1 '17 at 20:47
  • $\begingroup$ I agree--but I can imagine exceptions in the cases where $X$ is not of full rank. That's what suggested to me the situation isn't quite as trivial as it might appear to be. $\endgroup$ – whuber Nov 1 '17 at 21:05
  • 3
    $\begingroup$ imperial mate, not royal... :D (Nice answer, +1) $\endgroup$ – usεr11852 says Reinstate Monic Nov 2 '17 at 0:48
  • $\begingroup$ @usεr11852, I learned something today :) $\endgroup$ – Aksakal Nov 2 '17 at 13:58
2
$\begingroup$

An easy way to get this result is to remember that $\hat{y}$ is the projection of $y$ on the column space of $X.$ $\hat{\beta}$ is the vector of coefficients when $\hat{y}$ is expressed as a linear combination of the columns of $X$. If some column is scaled by a factor $c$, it is clear that the corresponding coefficient in the linear combination must be scaled by $1/c$.

Let $b_i$ be the values of $\hat{\beta}$ and $a_i$ be the values of the OLS solution when one column is scaled by $c.$ $$b_1x_1 + ... + b_ix_i + ...+ b_mx_m = a_1x_1 + ... a_i(cx_i) + ... +a_nx_n$$

implies that $b_j = a_j$ where $j \neq i$ and $b_i = a_ic$, assuming that the columns of $X$ are linearly independent.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.