I'm wondering if columns in the hidden layers can be forced to be orthogonal in a simple MLP.

A simple example : Let's say my input and output are : $X$ shape $(10,3)$ and $Y$ shape $(10,1)$.

With two weight matrix : $W_1$ shape $(3,2)$ and $W_2$ shape $(2,1)$

Lets define $V$ as $V = XW_1$ shape $(10,2)$

Is there a way to force both columns of $V$ to be orthogonal by adding a constraint ? How can this be done mathematically speaking in a neural network ?

V[:,0].dot(V[:,1].T) = 0



In your example the simplest solution is to factorize the inputs. Consider the orthgonality condition of yours: $$V'V=I$$ $$(XW_1)'XW_1=$$ $$W_1'X'XW_1=I$$ $$W_1W_1'X'XW_1W_1'=WW_1'$$ $$W_1W_1'X'X=I$$ $$W_1W_1'=(X'X)^{-1}$$

So, all you need to do in this case is to factorize the inverse of the covariance matrix $(X'X)^{-1}$ of inputs. This can be done with Cholesky decomposition, for instance.

So, if you have NN and want to do this to a layer other than the first, then you'd have to plug this procedure manually, maybe implement your own layer. I don't think you could do this with out of the box layers in popular frameworks.

A less elegant solution would be to impose this condition in the cost function. So, you'd store the $V$ matrix in cache, then when you calculate the cost function use its deviation from orthogonality as a component of the cost.

  • $\begingroup$ Very interesting answer, thank you. I'm gonna try the first method then the second one if I feel like it. Im not using a library to create my NN so I can implement it. $\endgroup$ – CDVC23 Mar 22 '18 at 18:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.