To what exactly does the term "activations" refer in neural networks?

Question

Does it refer to the input or the output of the activation function?

The literature seems to be inconsistent. A few examples:

Activations = Input of the activation function

• Deep Learning Book, Goodfellow et al., Pages 208, 209

  $a^{(k)} = b^{(k)} + W^{(k)}h^{(k-1)}$ [...] the activations $a^{(k)}$

• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift, Ioffe et al.

  We want to a preserve the information in the network, by normalizing the activations [...] Note that, since we normalize Wu+b, the bias b can be ignored ...

• http://cs231n.github.io/neural-networks-1/ (describing ReLU)

  this one is a common choice and simply thresholds all activations that are below zero to zero

Activations = Output of the activation function

• Binarized Neural Networks: Training Neural Networks with Weights and Activations Constrained to +1 or −1, Courbariaux et al speaks of pre-activations and activations

• http://cs231n.github.io/neural-networks-1/

  h1 = f(np.dot(W1, x) + b1) # calculate first hidden layer activations

  h2 = f(np.dot(W2, h1) + b2) # calculate second hidden layer activations

Answer 1

The simplest representation of a neural network is a Multi-Layer Perceptron or MLP. In its simplest form, MLP is just three layers.

An input layer represented by matrix $X \in \mathbb{R}^{N\times d}$ where $N$ is the number of training examples and $d$ is the number of features.

A hidden layer which is usually a ReLU or a logistic sigmoid function. Hidden layer $i$ could be a ReLU function which is represented by $$h_i(x) = \text{ReLU}(x) = max(x, 0)$$ In other words if the input to the ReLU function is negative, the function outputs a $0$. If the inputs x are positive, the ReLU function will output $x$.

The hidden layer feeds into the output layer which is just another function. This function could be squared error function (in the context of regression) or softmax (in the case of multiclass classification). The MLP is complete when you consider the weight and bias matrices but we don't need them for now.

Th activation function is just what the name suggests... a function. In the example above, the activation function for the hidden layer is the ReLU function. The activation function for the output layer was squared error or softmax.

When someone in Machine Learning uses the word $\text{activations}$, they are almost always referring to the output of the activation function. The possible activations in the hidden layer in the example above could only either be a $0$ or a $1$.

Note that the hidden activations (output from the hidden layer) could become input to other activation functions (in this case; the output layer activation functions). Pre-activation is the input to an activation function.

On a final note, I come from a statistics background which is a much older and more developed field. The notation in Statistics is pretty much standard. In machine learning however, the notation and the nomenclature are still evolving so I would not be surprised to see some authors use some terms differently. Context is your best friend when reading machine learning texts.

Answer 2

Personally, I would say that the neuron's activation is the output of the activation function. But there is clearly inconsistency in the literature, as you show. So how do I justify my intuition against highly-important published texts such as Goodfellow et al., 2016.

To make the case for why it should be the output, we can first draw from the nomenclature in neuroscience, as this is where we derive most of the terms from in the first place. For example, A. Hoffmann (2001), writes (quote found here):

Neural activations are mostly stimulated circularly. A neuron is activated by other neurons to which it is connected. In turn, its own activation stimulates other connected neurons to activation.

Here, clearly, the activation of the neuron is what stimulates antecedent neurons, i.e. it must be the output of the neuron.

Secondly, despite the naming of the mathematical notation, Goodfellow et al. also allude to the same conclusion in this passage (pp. 165):

Each unit resembles a neuron in the sense that it receives input from many other units and computes its own activation value.

Finally, in the passage that you cite from Goodfellow et al. (2016), they do make reference to $a^{(k)}$ as being the "pre-nonlinearity activation" (despite referring to it just as the "activation" in the algorithm description). The fact that this is being explicitly specified implies to me that the other "kind of activation" is the primary use of the term.