It may seem kind of silly, but can anyone please show me the intermediate steps implied by the second equality in this derivation?

$$e^\prime e = \left(y - Xb\right)^\prime\left(y - Xb\right) = y^\prime y - 2b^\prime X^\prime y + b^\prime X^\prime X b.$$

  • $\begingroup$ @statmerkur The theorem in that form does not apply to matrices because it requires multiplication to be commutative! $\endgroup$ – whuber Jan 19 '19 at 16:25
  • $\begingroup$ @statmerkur What you write simply is not the case for matrices larger than $1\times 1.$ What does hold is that $(x\pm y)^2 = x^2 \pm (xy + yx) + y^2.$ Asserting otherwise is both wrong and misleading. Moreover, it doesn't even apply here, because it ignores the transposition altogether. $\endgroup$ – whuber Jan 19 '19 at 17:07
  • $\begingroup$ @whuber I just wanted to give a hint to the fact that after transposing the first factor we have $(y^\prime -b^\prime X^\prime )(y-Xb)=y^\prime y-y^\prime Xb-b^\prime X^\prime y+b^\prime X^\prime Xb$ which in my eyes, as we are basically left with scalars here (as gunes says in his A), is very similar to what we have in $x^2 \pm 2xy + y^2$. I did not mean to state that the Binomal theorem holds for matrices. I see now that my comment was very imprecise and could be misleading - so I'm probably going to delete it. $\endgroup$ – statmerkur Jan 19 '19 at 17:56
  • $\begingroup$ @Peter would you care to be a little more specific about what isn't true about this statement? $\endgroup$ – whuber Jan 21 '19 at 19:01

Quite a few rules are actually needed for this derivation: almost all the algebraic (ring) axioms for real numbers, the axioms for real matrices as a (right) algebra over the real numbers, and properties of transposition as an anti-isomorphism of that algebra. The only numerical facts needed are used in the last step.

It may be of interest that no details of matrix multiplication are needed for this derivation, showing that it holds in more general statistical settings wherever the same axioms apply. For instance, real numbers can be replaced by complex numbers; transposition may be replaced by Hermitean conjugation; vectors may be replaced by (certain spaces of) functions and matrices by linear operators. All have applications, but there is no need to memorize or re-prove this result in all the different cases.

Here is a sketch of the details, assuming $e$ is a vector (as is evident from the form of the right hand side). (In a multivariate least squares model $e$ would be a matrix in its own right, but then a slightly different result would hold.)

  1. $(y-Xb)^\prime = y^\prime - (Xb)^\prime.$ Transposition is a linear operation.

  2. $(Xb)^\prime = b^\prime X^\prime.$ Transposition is an anti-isomorphism.

  3. $(y^\prime - b^\prime X^\prime)(y-Xb) = y^\prime (y-Xb) - (b^\prime X^\prime) (y-Xb).$ This is because $(B-C)A = B A - C A$ for any matrices $A,B,C$ where all operations are defined. Matrix multiplication is right-distributive. In the application, $A=y-Xb.$

  4. Similarly, $y^\prime (y-Xb) = y^\prime y - y^\prime (Xb)$ and $(b^\prime X^\prime)(y-Xb) = (b^\prime X^\prime)y - (b^\prime X^\prime)(Xb)$ because matrix multiplication is left-distributive.

  5. $y^\prime(Xb)$ is a $1\times 1$ matrix and therefore equals its own transpose, $y^\prime (Xb) = (y^\prime(Xb))^\prime.$

  6. As in $(2),$ $(y^\prime(Xb))^\prime = (Xb)^\prime (y^\prime)^\prime.$

  7. $(y^\prime)^\prime = y.$ Transposition is an involution.

  8. Applying $(2)$ and $(7)$ to $(5)$ and $(6)$ yields $(y^\prime (Xb))^\prime = (b^\prime X^\prime)y.$

  9. Substituting $(8)$ into the first term in $(4)$ and then substituting that result into $(3)$ produces $$(y-Xb)^\prime(y-Xb) = (y^\prime y - (b^\prime X^\prime) y) - ((b^\prime X^\prime)y - (b^\prime X^\prime)(Xb)).$$

  10. By definition, a matrix difference $A-B$ means the linear combination $(1)A + (-1)B.$ In light of this, the distributive, associative, and commutative laws of scalar multiplication (which parallel the same laws for multiplying real numbers) allow the right hand side of $(9)$ to be written $$y^\prime y - (1+1)(b^\prime X^\prime)y + (-1)(-1)(b^\prime X^\prime)(Xb).$$

  11. Several applications of the associative law of matrix multiplication show that parentheses in any sequence of matrix products can be dropped, because it doesn't matter how the products are grouped provided the order remains the same. Thus, for instance, $(b^\prime X^\prime)(Xb)$ may unambiguously be written $b^\prime X^\prime X b.$

  12. Finally, $1+1=2$ and $(-1)(-1)=1,$ whence $(10)$ can be simplified to $$y^\prime y - 2b^\prime X^\prime y + b^\prime X^\prime X b.$$

  • 1
    $\begingroup$ +10. Excellent detail. $\endgroup$ – StatsStudent Jan 19 '19 at 17:19


Then, note that each term here is scalar, and so $y^TXb$, i.e. ($(1\times n)(n \times k)(k \times 1)$), where $n$ is number of samples (data points), $k$ is number of regressors, including bias term.

For scalar terms, we can either take the transpose or not, i.e. $\alpha=\alpha^T$. So, $y^TXb=(y^TXb)^T=b^TX^Ty$, which is the third term above. Then, MSE becomes $y^Ty-2b^TX^Ty+b^TX^TXb$.

Note: If you give additional information about where you confronted this equation (e.g. linear regression lecture etc.), it'd better suit this forum; otherwise, this question may also suit to math forum (maybe better), although the procedure is quite common in ML for MSE calculation.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.