neural networks: can the activation of a neuron be thought of as a sort of angle between input and neuron?

Question

I heard that the activation could be thought of as a unnormalized angle between input and nodes.

How is this true?

My idea is that:

We have an input vector $\vec{x}$ per feature (?). This then gets multiplied with a dot product to a weight vector connecting the input to the neuron $\vec{w}$ and thus we have $\vec{x} \cdot \vec{w}$.

Now the dot product has to do with directions, such that it is minimized when they are opposite, and it is maximized when they are in the same direction. Thus it also has to do with the angle:

$\vec{x} \cdot \vec{w} = |\vec{x}| |\vec{w}| \cos{\theta}$

where $\theta$ si the angle between them. After the dot product we can take the $\tanh$ activation function and we would have $\tanh(\vec{x} \cdot \vec{w}) = \tanh(|\vec{x}| |\vec{w}| \cos{\theta})$

This is the further I could get. Any idea?

Also, is it correct that in the neural network we put a vector per each feature? I mean I know we should put an input neuron per each feature, so does that mean that each input neuron holds a vector? Or is the vector formulation a shortcut for the entire network?

Answer 1

Your first formula already shows that the activation (without applying the activation function) is equal to the unnormalized (cosine of the) angle. I think that's pretty much how far this goes actually. This is the activation per node.

Each node has a couple of weights, that correspond to the activation in the previous layer. So if there are 8 nodes in the previous layer, a node in the next layer has 8 weights associated with it. The activation of this node is the activation function applied to the unnormalized angle between these weights and the activations in the previous layer.