I was reading the FaceNet paper and in the 3rd paragraph of the introduction it says:
Previous face recognition approaches based on deep networks use a classification layer trained over a set of known face identities and then take an intermediate bottleneck layer as a representation used to generalize recognition beyond the set of identities used in training.
I was wondering what they mean by an intermediate bottleneck layer?
A bottleneck layer is a layer that contains few nodes compared to the previous layers. It can be used to obtain a representation of the input with reduced dimensionality. An example of this is the use of autoencoders with bottleneck layers for nonlinear dimensionality reduction.
My understanding of the quote is that previous approaches use a deep network to classify faces. They then take the first several layers of this network, from the input up to some intermediate layer (say, the $k$th layer, containing $n_k$ nodes). This subnetwork implements a mapping from the input space to an $n_k$-dimensional vector space. The $k$th layer is a bottleneck layer, so the vector of activations of nodes in the $k$th layer gives a lower dimensional representation of the input. The original network can't be used to classify new identities, on which it wasn't trained. But, the $k$th layer may provide a good representation of faces in general. So, to learn new identities, new classifier layers can be stacked on top of the $k$th layer and trained. Or, the new training data can be fed through the subnetwork to obtain representations from the $k$th layer, and these representations can be fed to some other classifier.
Adding to the previous answer: Bottlenecks can also be understood as a design pattern, consisting of three convolution layers, introduced by the ResNet paper.
Deeper Bottleneck Architectures. Next, we describe our deeper nets for ImageNet. Because of concerns on the training time that we can afford, we modify the building block as a bottleneck. For each residual function F , we use a stack of 3 layers instead of 2 (Fig. 5). The three layers are 1×1, 3×3, and 1×1 convolutions, where the 1×1 layers are responsible for reducing and then increasing (restoring) dimensions, leaving the 3x3 layer a bottleneck with smaller input/output dimensions. Fig 5. shows an example, where both designs have similar time complexity.
