# Properties of hidden layers in neural network

I'm wondering if columns in the hidden layers can be forced to be orthogonal in a simple MLP.

A simple example : Let's say my input and output are : $X$ shape $(10,3)$ and $Y$ shape $(10,1)$.

With two weight matrix : $W_1$ shape $(3,2)$ and $W_2$ shape $(2,1)$

Lets define $V$ as $V = XW_1$ shape $(10,2)$

Is there a way to force both columns of $V$ to be orthogonal by adding a constraint ? How can this be done mathematically speaking in a neural network ?

V[:,0].dot(V[:,1].T) = 0


Thanks

In your example the simplest solution is to factorize the inputs. Consider the orthgonality condition of yours: $$V'V=I$$ $$(XW_1)'XW_1=$$ $$W_1'X'XW_1=I$$ $$W_1W_1'X'XW_1W_1'=WW_1'$$ $$W_1W_1'X'X=I$$ $$W_1W_1'=(X'X)^{-1}$$
So, all you need to do in this case is to factorize the inverse of the covariance matrix $(X'X)^{-1}$ of inputs. This can be done with Cholesky decomposition, for instance.
A less elegant solution would be to impose this condition in the cost function. So, you'd store the $V$ matrix in cache, then when you calculate the cost function use its deviation from orthogonality as a component of the cost.