I'm wondering if columns in the hidden layers can be forced to be orthogonal in a simple MLP.
A simple example : Let's say my input and output are : $X$ shape $(10,3)$ and $Y$ shape $(10,1)$.
With two weight matrix : $W_1$ shape $(3,2)$ and $W_2$ shape $(2,1)$
Lets define $V$ as $V = XW_1$ shape $(10,2)$
Is there a way to force both columns of $V$ to be orthogonal by adding a constraint ? How can this be done mathematically speaking in a neural network ?
V[:,0].dot(V[:,1].T) = 0