Does it refer to the input or the output of the activation function?

The literature seems to be inconsistent. A few examples:

Activations = Input of the activation function

Activations = Output of the activation function


Personally, I would say that the neuron's activation is the output of the activation function. But there is clearly inconsistency in the literature, as you show. So how do I justify my intuition against highly-important published texts such as Goodfellow et al., 2016.

To make the case for why it should be the output, we can first draw from the nomenclature in neuroscience, as this is where we derive most of the terms from in the first place. For example, A. Hoffmann (2001), writes (quote found here):

Neural activations are mostly stimulated circularly. A neuron is activated by other neurons to which it is connected. In turn, its own activation stimulates other connected neurons to activation.

Here, clearly, the activation of the neuron is what stimulates antecedent neurons, i.e. it must be the output of the neuron.

Secondly, despite the naming of the mathematical notation, Goodfellow et al. also allude to the same conclusion in this passage (pp. 165):

Each unit resembles a neuron in the sense that it receives input from many other units and computes its own activation value.

Finally, in the passage that you cite from Goodfellow et al. (2016), they do make reference to $a^{(k)}$ as being the "pre-nonlinearity activation" (despite referring to it just as the "activation" in the algorithm description). The fact that this is being explicitly specified implies to me that the other "kind of activation" is the primary use of the term.

  • 1
    $\begingroup$ (+1) I have a dislike of downvotes that do not offer any alternatives and see nothing blatantly wrong here. $\endgroup$ – Carl Dec 8 '18 at 12:58
  • $\begingroup$ I can't speak for the downvoter, but instead of just stating that it's the output, I would prefer additional references that back up this claim. $\endgroup$ – Kilian Batzner Nov 15 '20 at 12:22
  • $\begingroup$ Also, the counter-examples aren't just omitting the function for brevity. Goodfellow2016 is a comprehensive book with a well-considered and consistent notation. As for the batch normalization paper, it does matter which activations you normalize, see reddit.com/r/MachineLearning/comments/67gonq/… $\endgroup$ – Kilian Batzner Nov 15 '20 at 12:26

The simplest representation of a neural network is a Multi-Layer Perceptron or MLP. In its simplest form, MLP is just three layers.

An input layer represented by matrix $X \in \mathbb{R}^{N\times d}$ where $N$ is the number of training examples and $d$ is the number of features.

A hidden layer which is usually a ReLU or a logistic sigmoid function. Hidden layer $i$ could be a ReLU function which is represented by $$h_i(x) = \text{ReLU}(x) = max(x, 0)$$ In other words if the input to the ReLU function is negative, the function outputs a $0$. If the inputs x are positive, the ReLU function will output $x$.

The hidden layer feeds into the output layer which is just another function. This function could be squared error function (in the context of regression) or softmax (in the case of multiclass classification). The MLP is complete when you consider the weight and bias matrices but we don't need them for now.

Th activation function is just what the name suggests... a function. In the example above, the activation function for the hidden layer is the ReLU function. The activation function for the output layer was squared error or softmax.

When someone in Machine Learning uses the word $\text{activations}$, they are almost always referring to the output of the activation function. The possible activations in the hidden layer in the example above could only either be a $0$ or a $1$.

Note that the hidden activations (output from the hidden layer) could become input to other activation functions (in this case; the output layer activation functions). Pre-activation is the input to an activation function.

On a final note, I come from a statistics background which is a much older and more developed field. The notation in Statistics is pretty much standard. In machine learning however, the notation and the nomenclature are still evolving so I would not be surprised to see some authors use some terms differently. Context is your best friend when reading machine learning texts.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.