mathematical definitions of neural network terminology: layer, hidden unit, neuron

Question

I think about feedforward neural networks from a mathematical perspective. There is an input array $x_0$ (often an image) that is passed through several composed functions to arrive at an output. For example, the array might be passed through a convolution function $\text{conv}_1(\cdot)$, a ReLU function $\text{relu}(\cdot)$, another convolution function $\text{conv}_2(\cdot)$, another ReLU function, then a max pooling function $\text{maxpool}(\cdot)$, etc. The whole network might look like this: \begin{align} x_1 &= \text{conv}_1(x_0) \\ x_2 &= \text{relu}(x_1) \\ x_3 &= \text{conv}_2(x_2) \\ x_4 &= \text{relu}(x_3) \\ x_5 &= \text{maxpool}(x_4) \\ &\vdots \end{align}

Despite reading many sources on neural networks, I am confused as to what the terms layer, hidden unit, and neuron refer to. For example, does "layer" refer to a function, or the arrays themselves? So is the "first layer" in the network above the $\text{conv}_1(\cdot)$ function, the $\text{relu}(\text{conv}_1(\cdot))$ function, the $\text{x}_0$ array, or the $\text{x}_1$ array? I have a similar confusion for the terms "hidden unit" and "neuron".

Answer 1

Firstly, the term "layer" I suppose was first introduced when describing fully connected feedforward networks, which have a simpler structure than what you're talking about. In general, I think "layer" usually means the following. Basically, a neural net can be expressed as a sequence of transformations of the following form: $x_{i+1}=f(w_i°x_i)$, where $w_i$ is a weight vector and $f$ is an arbitrary transformation, and $w°x$ denotes some operation on the two vectors, such as convolution or matrix multiplication (I'm ignoring recurrent weights and inputs here for simplicity). In your example, $f$ can be simply an activation function or an activation function followed by pooling, etc. With this nomenclature, $i$ indexes the layers ($i=0$ is the input let's say), and a layer would correspond to the transformation $f(w_i°x_i)$. At least that's how I see it.

A hidden layer is any layer with $i>0$ in the above nomenclature. A hidden unit is basically a component of $f(w_i°x_i)$. A neuron is another name for hidden unit.

Answer 2

A "layer" refers to a particular node (or connected set of nodes) in the "computation graph" (a computation graph is a graph describing a computation where each node corresponds to an operation, such as addition, multiplcation, convolution, etc, with edges to denote how inputs and outputs flow between nodes of the graph).

It doesn't refer to the function itself (for example, you might use the convolution function many times in a network, in which case you probably have more than just one layer -- it would be weird to call all those convolutions one layer simply because it's the same pure function).

It doesn't refer to the "arrays themselves" (i'm assuming you mean the outputs of the convolutions). Those are called "feature maps". It wouldn't make much sense to call these layers, because what if you convolve two different images with the same weight matrix and get two different outputs? Surely, you still have one layer.

As for whether the layer consists of just the convolution, the conv and relu, or the conv, a relu, and a batchnorm, etc -- well that's just convention. I think typically conv and relu (and optionally batch/layer/instance/group/spectral norm) together are considered one layer.

As a bonus term, you'll also commonly see "block", "module", or "subnetwork" used to describe assemblies of many layers into an even larger computation, with several blocks being used to build up the full network.

A "neuron" is probably most often used to describe the computation of a single value in the output of any given layer. I say it's "the computation" and not the value itself, for similar reasons to why there's a difference between a "layer" and a "feature map". Most commonly, "neuron" implies post-activation (i.e. after a conv and activation), but not always.

A hidden-unit is a neuron which is not a root or sink of the computation graph.

Anyway, for the most part these concepts are meant to be taken informally, and if it is critical exactly what they mean, it's usually clear from context.