Return to Answer

Here is a more formal answer (more elegant proofs that start with something like "consider the space spanned by the columns of $X$..." are surely possible) to the question of why just changing the order of the regressors does not matter. As I sketch at the end, the question you link to deals with a case where more than just permuting the columns is happening (as the accepted answere there, I believe, also explains quite well).

Consider changing the position of the variables in the $(n\times p)$ regressor matrix $$ X=(X_1,\ldots,X_p), $$ where $X_j=(x_{1j},\ldots,x_{nj})'$, $j=1,\ldots,p$, amounts to postmultiplying $X$ with a $(p\times p)$ permutation matrix $P$ that has a single entry 1 in each column $j$ that indicates the new column position of that regressor $X_j$.

For example, if the new columns are to be the old columns 2, 1 and 3, we have $$P=\begin{pmatrix} 0&1&0\\ 1&0&0\\ 0&0&1 \end{pmatrix}$$

This matrix $P$ is invertible, being just a permuted version of the identity matrix.

Also, as is easy to check that $P'P=I$, $P^{-1}$ is equal to the transpose of $P$, $P^{-1}=P'$.

Thus, the OLS coefficient of the regression of $y$ on the transformed regressors, call it $\hat\beta_t$, is \begin{eqnarray*} \hat\beta_t&=&((XP)'XP)^{-1}(XP)'y\\ &\stackrel{(AB)'=B'A'}{=}&(P'X'XP)^{-1}P'X'y\\ &\stackrel{(ABC)^{-1}=C^{-1}B^{-1}A^{-1}}{=}&P^{-1}(X'X)^{-1}\underbrace{(P')^{-1}P'}_{=I}X'y\\ &=&P^{-1}(X'X)^{-1}X'y\\ &=&P'(X'X)^{-1}X'y\\ &=&P'\hat\beta\\ \end{eqnarray*} Here, $P'$ is a matrix that permutes the row elements of $\hat\beta$, and hence permutes the entries of the original coeffient estimator $\hat\beta$ according to the permutation of the columns.

To see why the question you link to addresses a slightly different situation in which something else happens than just permuting the columns, I suggest to inspect model.matrix(out_1) and model.matrix(out_2) in that code, which gives you the different regressor matrices in the two models.

Here is a more formal answer (more elegant proofs that start with something like "consider the space spanned by the columns of $X$..." are surely possible) to the question of why just changing the order of the regressors does not matter. As I sketch at the end, the question you link to deals with a case where more than just permuting the columns is happening (as the accepted answer there, I believe, also explains quite well).

Consider changing the position of the variables in the $(n\times p)$ regressor matrix $$ X=(X_1,\ldots,X_p), $$ where $X_j=(x_{1j},\ldots,x_{nj})'$, $j=1,\ldots,p$, amounts to postmultiplying $X$ with a $(p\times p)$ permutation matrix $P$ that has a single entry 1 in each column $j$ that indicates the new column position of that regressor $X_j$.

For example, if the new columns are to be the old columns 2, 1 and 3, we have $$P=\begin{pmatrix} 0&1&0\\ 1&0&0\\ 0&0&1 \end{pmatrix}$$

This matrix $P$ is invertible, being just a permuted version of the identity matrix.

Also, as it is easy to check that $P'P=I$, $P^{-1}$ is equal to the transpose of $P$, $P^{-1}=P'$.

Thus, the OLS coefficient of the regression of $y$ on the transformed regressors, call it $\hat\beta_t$, is \begin{eqnarray*} \hat\beta_t&=&((XP)'XP)^{-1}(XP)'y\\ &\stackrel{(AB)'=B'A'}{=}&(P'X'XP)^{-1}P'X'y\\ &\stackrel{(ABC)^{-1}=C^{-1}B^{-1}A^{-1}}{=}&P^{-1}(X'X)^{-1}\underbrace{(P')^{-1}P'}_{=I}X'y\\ &=&P^{-1}(X'X)^{-1}X'y\\ &=&P'(X'X)^{-1}X'y\\ &=&P'\hat\beta\\ \end{eqnarray*} Here, $P'$ is a matrix that permutes the row elements of $\hat\beta$, and hence permutes the entries of the original coeffient estimator $\hat\beta$ according to the permutation of the columns.

To see why the question you link to addresses a slightly different situation in which something else happens than just permuting the columns, I suggest to inspect model.matrix(out_1) and model.matrix(out_2) in that code, which gives you the different regressor matrices in the two models.

To add a formal answer (more elegant proofs that start with something like "consider the space spanned by the columns of $X$..." are surely possible) to the question, changing the position of the variables in the $(n\times p)$ regressor matrix $$ X=(X_1,\ldots,X_p), $$ where $X_j=(x_{1j},\ldots,x_{nj})'$, $j=1,\ldots,p$, amounts to postmultiplying $X$ with a $(p\times p)$ permutation matrix $P$ that has a single entry 1 in each column $j$ that indicates the new column position of that regressor $X_j$.

For example, if the new columns are to be the old columns 2, 1 and 3, we have $$P=\begin{pmatrix} 0&1&0\\ 1&0&0\\ 0&0&1 \end{pmatrix}$$

This matrix $P$ is invertible, being just a permuted version of the identity matrix.

Also, as is easy to check that $P'P=I$, $P^{-1}$ is equal to the transpose of $P$, $P^{-1}=P'$.

Thus, the OLS coefficient of the regression of $y$ on the transformed regressors, call it $\hat\beta_t$, is \begin{eqnarray*} \hat\beta_t&=&((XP)'XP)^{-1}(XP)'y\\ &\stackrel{(AB)'=B'A'}{=}&(P'X'XP)^{-1}P'X'y\\ &\stackrel{(ABC)^{-1}=C^{-1}B^{-1}A^{-1}}{=}&P^{-1}(X'X)^{-1}\underbrace{(P')^{-1}P'}_{=I}X'y\\ &=&P^{-1}(X'X)^{-1}X'y\\ &=&P'(X'X)^{-1}X'y\\ &=&P'\hat\beta\\ \end{eqnarray*} Here, $P'$ is a matrix that permutes the row elements of $\hat\beta$, and hence permutes the entries of the original coeffient estimator $\hat\beta$ according to the permutation of the columns.

To see why the question you link to addresses a slightly different situation in which something else happens than just permuting the columns, I suggest to inspect model.matrix(out_1) and model.matrix(out_2) in that code, which gives you the different regressor matrices in the two models.

Here is a more formal answer (more elegant proofs that start with something like "consider the space spanned by the columns of $X$..." are surely possible) to the question of why just changing the order of the regressors does not matter. As I sketch at the end, the question you link to deals with a case where more than just permuting the columns is happening (as the accepted answere there, I believe, also explains quite well).

Consider changing the position of the variables in the $(n\times p)$ regressor matrix $$ X=(X_1,\ldots,X_p), $$ where $X_j=(x_{1j},\ldots,x_{nj})'$, $j=1,\ldots,p$, amounts to postmultiplying $X$ with a $(p\times p)$ permutation matrix $P$ that has a single entry 1 in each column $j$ that indicates the new column position of that regressor $X_j$.

For example, if the new columns are to be the old columns 2, 1 and 3, we have $$P=\begin{pmatrix} 0&1&0\\ 1&0&0\\ 0&0&1 \end{pmatrix}$$

This matrix $P$ is invertible, being just a permuted version of the identity matrix.

Also, as is easy to check that $P'P=I$, $P^{-1}$ is equal to the transpose of $P$, $P^{-1}=P'$.

Thus, the OLS coefficient of the regression of $y$ on the transformed regressors, call it $\hat\beta_t$, is \begin{eqnarray*} \hat\beta_t&=&((XP)'XP)^{-1}(XP)'y\\ &\stackrel{(AB)'=B'A'}{=}&(P'X'XP)^{-1}P'X'y\\ &\stackrel{(ABC)^{-1}=C^{-1}B^{-1}A^{-1}}{=}&P^{-1}(X'X)^{-1}\underbrace{(P')^{-1}P'}_{=I}X'y\\ &=&P^{-1}(X'X)^{-1}X'y\\ &=&P'(X'X)^{-1}X'y\\ &=&P'\hat\beta\\ \end{eqnarray*} Here, $P'$ is a matrix that permutes the row elements of $\hat\beta$, and hence permutes the entries of the original coeffient estimator $\hat\beta$ according to the permutation of the columns.

To see why the question you link to addresses a slightly different situation in which something else happens than just permuting the columns, I suggest to inspect model.matrix(out_1) and model.matrix(out_2) in that code, which gives you the different regressor matrices in the two models.

To add a formal answer to the question, changing the position of the variables in the $(n\times p)$ regressor matrix $$ X=(X_1,\ldots,X_p), $$ where $X_j=(x_{1j},\ldots,x_{nj})'$, $j=1,\ldots,p$, amounts to postmultiplying $X$ with a $(p\times p)$ permutation matrix $P$ that has a single entry 1 in each column $j$ that indicates the new column position of that regressor $X_j$.

For example, if the new columns are to be the old columns 2, 1 and 3, we have $$P=\begin{pmatrix} 0&1&0\\ 1&0&0\\ 0&0&1 \end{pmatrix}$$

This matrix $P$ is invertible, being just a permuted version of the identity matrix.

Also, as is easy to check that $P'P=I$, $P^{-1}$ is equal to the transpose of $P$, $P^{-1}=P'$.

Thus, the OLS coefficient of the regression of $y$ on the transformed regressors, call it $\hat\beta_t$, is \begin{eqnarray*} \hat\beta_t&=&((XP)'XP)^{-1}(XP)'y\\ &\stackrel{(AB)'=B'A'}{=}&(P'X'XP)^{-1}P'X'y\\ &\stackrel{(ABC)^{-1}=C^{-1}B^{-1}A^{-1}}{=}&P^{-1}(X'X)^{-1}\underbrace{(P')^{-1}P'}_{=I}X'y\\ &=&P^{-1}(X'X)^{-1}X'y\\ &=&P'(X'X)^{-1}X'y\\ &=&P'\hat\beta\\ \end{eqnarray*} Here, $P'$ is a matrix that permutes the row elements of $\hat\beta$, and hence permutes the entries of the original coeffient estimator $\hat\beta$ according to the permutation of the columns.

To add a formal answer (more elegant proofs that start with something like "consider the space spanned by the columns of $X$..." are surely possible) to the question, changing the position of the variables in the $(n\times p)$ regressor matrix $$ X=(X_1,\ldots,X_p), $$ where $X_j=(x_{1j},\ldots,x_{nj})'$, $j=1,\ldots,p$, amounts to postmultiplying $X$ with a $(p\times p)$ permutation matrix $P$ that has a single entry 1 in each column $j$ that indicates the new column position of that regressor $X_j$.

For example, if the new columns are to be the old columns 2, 1 and 3, we have $$P=\begin{pmatrix} 0&1&0\\ 1&0&0\\ 0&0&1 \end{pmatrix}$$

This matrix $P$ is invertible, being just a permuted version of the identity matrix.

Also, as is easy to check that $P'P=I$, $P^{-1}$ is equal to the transpose of $P$, $P^{-1}=P'$.

Thus, the OLS coefficient of the regression of $y$ on the transformed regressors, call it $\hat\beta_t$, is \begin{eqnarray*} \hat\beta_t&=&((XP)'XP)^{-1}(XP)'y\\ &\stackrel{(AB)'=B'A'}{=}&(P'X'XP)^{-1}P'X'y\\ &\stackrel{(ABC)^{-1}=C^{-1}B^{-1}A^{-1}}{=}&P^{-1}(X'X)^{-1}\underbrace{(P')^{-1}P'}_{=I}X'y\\ &=&P^{-1}(X'X)^{-1}X'y\\ &=&P'(X'X)^{-1}X'y\\ &=&P'\hat\beta\\ \end{eqnarray*} Here, $P'$ is a matrix that permutes the row elements of $\hat\beta$, and hence permutes the entries of the original coeffient estimator $\hat\beta$ according to the permutation of the columns.

To see why the question you link to addresses a slightly different situation in which something else happens than just permuting the columns, I suggest to inspect model.matrix(out_1) and model.matrix(out_2) in that code, which gives you the different regressor matrices in the two models.