neural networks: can the activation of a neuron be thought of as a sort of angle between input and neuron?

I heard that the activation could be thought of as a unnormalized angle between input and nodes.

How is this true?

My idea is that:

We have an input vector $\vec{x}$ per feature (?). This then gets multiplied with a dot product to a weight vector connecting the input to the neuron $\vec{w}$ and thus we have $\vec{x} \cdot \vec{w}$.

Now the dot product has to do with directions, such that it is minimized when they are opposite, and it is maximized when they are in the same direction. Thus it also has to do with the angle:

$\vec{x} \cdot \vec{w} = |\vec{x}| |\vec{w}| \cos{\theta}$

where $\theta$ si the angle between them. After the dot product we can take the $\tanh$ activation function and we would have $\tanh(\vec{x} \cdot \vec{w}) = \tanh(|\vec{x}| |\vec{w}| \cos{\theta})$

This is the further I could get. Any idea?

Also, is it correct that in the neural network we put a vector per each feature? I mean I know we should put an input neuron per each feature, so does that mean that each input neuron holds a vector? Or is the vector formulation a shortcut for the entire network?

• Thank you so much. That would come from my formula by dividing through, right? $\cos(\theta) = \frac{\vec{x} \cdot \vec{w}}{|\vec{x}||\vec{w}|}$ so that only the dot product is not normalized! – Euler_Salter Oct 17 '17 at 12:13
• Yes, exactly, the norms of x and w are still in there. – Gijs Oct 17 '17 at 12:56