I heard that the activation could be thought of as a unnormalized angle between input and nodes.

How is this true?

My idea is that:

We have an input vector $\vec{x}$ per feature (?). This then gets multiplied with a dot product to a weight vector connecting the input to the neuron $\vec{w}$ and thus we have $\vec{x} \cdot \vec{w}$.

Now the dot product has to do with directions, such that it is minimized when they are opposite, and it is maximized when they are in the same direction. Thus it also has to do with the angle:

$\vec{x} \cdot \vec{w} = |\vec{x}| |\vec{w}| \cos{\theta}$

where $\theta$ si the angle between them. After the dot product we can take the $\tanh$ activation function and we would have $\tanh(\vec{x} \cdot \vec{w}) = \tanh(|\vec{x}| |\vec{w}| \cos{\theta})$

This is the further I could get. Any idea?

Also, is it correct that in the neural network we put a vector per each feature? I mean I know we should put an input neuron per each feature, so does that mean that each input neuron holds a vector? Or is the vector formulation a shortcut for the entire network?


1 Answer 1


Your first formula already shows that the activation (without applying the activation function) is equal to the unnormalized (cosine of the) angle. I think that's pretty much how far this goes actually. This is the activation per node.

Each node has a couple of weights, that correspond to the activation in the previous layer. So if there are 8 nodes in the previous layer, a node in the next layer has 8 weights associated with it. The activation of this node is the activation function applied to the unnormalized angle between these weights and the activations in the previous layer.

  • $\begingroup$ oh right, that was so straightforward! $\endgroup$ Oct 17, 2017 at 12:12
  • $\begingroup$ Thank you so much. That would come from my formula by dividing through, right? $\cos(\theta) = \frac{\vec{x} \cdot \vec{w}}{|\vec{x}||\vec{w}|}$ so that only the dot product is not normalized! $\endgroup$ Oct 17, 2017 at 12:13
  • $\begingroup$ Yes, exactly, the norms of x and w are still in there. $\endgroup$
    – Gijs
    Oct 17, 2017 at 12:56

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.