# How does the activation of neurons work?

I would need to understand better how the activation of the neurons in a neural network works. Let's suppose to work with Tanh (ranging in [-1,1]). I know that a neuron must be able to choose whether to accept the input from another neuron or not. But how is this process related to the activation function? If the value of Tanh for a certain neuron is close to -1, is that neuron switched off or is it still actively involved in the training? (And if it is actively involved, then why do we talk of "activation"?)

Also, I'm using h2o for deep learning anomaly detection. How do I choose between Tanh and Rectified Linear? I tried both and the outputs differ a lot...

• These neurons aren't really "on" or "off." As you note, in these cases the outputs vary over some range.
– Sycorax
Dec 9, 2016 at 15:25

In my opinion, understanding these things by analogy might be helpful in the beginning when you're first starting, but eventually you'll get stuck if you don't begin to understand these models on their own terms.

A neural network is a linear model (or a multinomial logit if classification) fitting the response as a function of derived variables:

$$\begin{array}{l l} y_{i} = \gamma_1 + \mathbf{V}^1_{i}\Gamma^1 + \epsilon_{i}\\ \mathbf{V}^1_{i} = a\left([\mathbf{1,V}^2_{i}]\mathbf{\Gamma}^2\right)\\ \mathbf{V}^2_{i} = a\left([\mathbf{1,V}^3_{i}]\mathbf{\Gamma}^3\right)\\ \hspace{1.5cm}\vdots\\ \mathbf{V}^L_{i} = a\left([\mathbf{1,Z}_{i}]\mathbf{\Gamma}^L\right)\\ \end{array}$$

The derived variables are the $V$ terms, and the $\Gamma$ are parameters. (The 1's are for the "bias" terms, which are analogous to intercepts in linear models.) Down on the bottom layer, your data $Z$ are transformed first by the parameters and then by the activation function into the first set of derived variables. The process continues upwards until you reach the top layer.

So to your question about activation functions -- the activation function ($a()$) simply transforms the linear combination of derived variables. It maps the real line to a subset of it. A hyperbolic tangent maps $V\Gamma$ to the interval $[-1, 1]$, and a RELU makes negative values of $V\Gamma$ go to zero. Which one makes the most sense is probably dataset-dependent.

And if $a(V\Gamma) = -1$, that simply means that one of your regressors at the top level takes a value of -1. I suppose that it is only "off" if it takes a value of zero.

And to your question of how to choose between different models -- the standard answer is "cross validation"

I think you're talking about perceptrons, not neurons. Perceptrons jump between discrete states, such as 0 and 1 when an input crosses the threshold. Neuron can produce continuous output on continuous inputs. So, your neuron with inverse tanh function accepts inputs between -1 and 1 and produces the output from $-\infty$ to $\infty$. For instance, inverse tanh of 0 is 0.