# Properties of hidden layers in neural network

I'm wondering if columns in the hidden layers can be forced to be orthogonal in a simple MLP.

A simple example : Let's say my input and output are : $X$ shape $(10,3)$ and $Y$ shape $(10,1)$.

With two weight matrix : $W_1$ shape $(3,2)$ and $W_2$ shape $(2,1)$

Lets define $V$ as $V = XW_1$ shape $(10,2)$

Is there a way to force both columns of $V$ to be orthogonal by adding a constraint ? How can this be done mathematically speaking in a neural network ?

V[:,0].dot(V[:,1].T) = 0


Thanks

## 1 Answer

In your example the simplest solution is to factorize the inputs. Consider the orthgonality condition of yours: $$V'V=I$$ $$(XW_1)'XW_1=$$ $$W_1'X'XW_1=I$$ $$W_1W_1'X'XW_1W_1'=WW_1'$$ $$W_1W_1'X'X=I$$ $$W_1W_1'=(X'X)^{-1}$$

So, all you need to do in this case is to factorize the inverse of the covariance matrix $(X'X)^{-1}$ of inputs. This can be done with Cholesky decomposition, for instance.

So, if you have NN and want to do this to a layer other than the first, then you'd have to plug this procedure manually, maybe implement your own layer. I don't think you could do this with out of the box layers in popular frameworks.

A less elegant solution would be to impose this condition in the cost function. So, you'd store the $V$ matrix in cache, then when you calculate the cost function use its deviation from orthogonality as a component of the cost.

• Very interesting answer, thank you. I'm gonna try the first method then the second one if I feel like it. Im not using a library to create my NN so I can implement it. – CDVC23 Mar 22 '18 at 18:17