There is a question here on Stackexchange- Does the order of explanatory variables matter when calculating their regression coefficients?

Although I understand the details here, how can we formally prove that the order of variables doesn't change the coefficient estimates? I was trying to use partitioned matrices, and just showing that by changing the order of partitions and the respective coefficient estimates matrix, I get the same result. However, it doesn't look like a formal proof. Can anyone help?

  • 2
    $\begingroup$ No proof is needed when you adopt the geometric point of view: the solution is the projection onto a subspace. This description makes no reference to the order of variables. If you want to be formal about it, this argument comes down to a linear algebraic result which is so trivial you may have a hard time finding it in any textbook: whenever $E=(e_1,e_2,\ldots, e_n)$ is a basis of a vector space, then so is $E^\prime=(e_{\sigma(1)}, e_{\sigma(2)}, \ldots, e_{\sigma(n)})$ for any permutation $\sigma$. $\endgroup$ – whuber Jan 16 '18 at 16:46

Let $P$ be a permutation matrix such that $PP^T = P^TP = I$. It can be shown $PA = AP$ for any symmetric matrix $A$.

Then the permuted linear estimator $Y = P \mathbf{X}\beta $ has method of moments estimator $\hat{\beta}_P =\left( \mathbf{X}^T P^TP \mathbf{X} \right)^{-1} P\mathbf{X}^TY = P \left( \mathbf{X}^T \mathbf{X}\right)^{-1} \mathbf{X}^TY$ which is just the permutation of the unpermuted least squares estimator.

  • 1
    $\begingroup$ I think it's even more clear using the optimization problem LR solves. $\endgroup$ – Matthew Drury Jan 16 '18 at 17:41
  • $\begingroup$ There may be more than one "answer" to the problem. Mine can just be the elementary proof. $\endgroup$ – AdamO Jan 16 '18 at 17:45
  • $\begingroup$ The disadvantage of this elementary proof is that it applies only to full-rank problems. Another disadvantage is that it proves something other than what was asked: you have changed the order of observations, not the order of variables! (Moreover, you neglected to permute the responses $Y$ in parallel, which is necessary to obtain the correct result.) Finally, your assertion that permutation matrices commute with all symmetric matrices is false. There are simple counterexamples with $3\times 3$ matrices. (Bear in mind the symmetric group $\mathfrak{S}_n$ is not commutative for $n\ge 3$.) $\endgroup$ – whuber Jan 16 '18 at 19:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.