Consider an input layer $x$ ($D$ dimensional) , a hidden layer $y$ $K<D$ dimensional) and an output layer $z$ ($D$ dimensional) . Let the input to hidden matrix be $P$ and the hidden to output matrix be $Q$.

If we are using the neural network as an auto encoder then the error is the squared difference between $z$ and $x$. It can be shown that once the weights have converged , we can express $P$ and $Q$ as

$P=(Q^TQ)^{-1}Q^T$ and $Q=C_xP^T(PC_xP^T)^{-1}$

where $C_x=\displaystyle\sum_n x_nx_n^T$ and $n$ indexes the data points.

Question: If $Q$ has rank $K$, then for any invertible $K \times K$ matrix $A$, the matrices $P_*=AP$ and $Q_*=QA^{-1}$ also minimise the error. How do I show that we can always find a matrix $A$ such that $Q_*^TQ_*=\mathbb{I}$ and $P_*=Q_*^T$?


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.