Despite having dealt with Machine Learning for a few years now, I still find myself sporadically confused with elementary terminology, especially when it comes to Convolutional Neural Networks, and I've never found a source that clarifies them to full satisfaction.
This was originally more of a question, but I've ended up mostly answering it myself by writing down thoughts, so I guess now my questions are, "is this all correct?" and "if so, which meaning of 'Feature' is the most common?"
Take as our running example Layer 4 of Resnet-18, for which we have
- Input: $14 \times 14 \times 256$ (that's $\text{height} \times \text{width} \times \text{n_channels}$)
- Output: $7 \times 7 \times 512$
Where the operation is performed with $\text{kernel size}=(3,3)$, $\text{stide}=2$, $\text{padding}=0$. Here is a list of all related terms that I'd like to tease out:
- Filter, Kernel, Feature, Feature Map, Neuron
Here is my current attempt at doing so. The first three are relatively clear, then it gets confusing.
- A Filter seems to be a stack of 256 sliding windows that determine how the values in one of the channels of Layer #4's output are computed out Layer #4's input. Thus, there are $512$ Filters, each having $(3 \cdot 3) \cdot 256$ parameters. (Here, the only unclear part is whether the bias parameter would count as part of the Filter, but I mean that's really a detail.)
- The term Kernel (according to this answer) either refers to the same thing, or to just one of the Sliding windows. Thus, there are either $512$ Kernels or $512 \cdot 256$ Kernels. I don't know which of the two is more common.
- The term Feature Map seems to refer to the set of results in one output channel. Thus, there are $512$ Feature Maps.
- As for Feature, I can see at least three possible definitions, and I can't tell which one people usually refer to:
- The same as Feature Map; thus there would be $512$ Features.
- The output of one of the $7 \cdot 7$ elements in a Feature Map. In this case, the Output of Layer #4 would be identical to the set of Features. There would then be $512 \cdot 7 \cdot 7$ Features.
- The abstract computation implemented by a Filter. In this case, there would be 512 Features, one for each Feature Map. However, the Feature would refer to the computation, whereas the Feature Map would refer to the output. Thus, you could say that "Feature 412 had value 0.7352 on position $5 \times 3$", whereas the same sentence wouldn't work if you substituted "Feature Map". Rather, for each Feature, its Feature Map is what determines the values taken by the Feature.
- Additionally, there is a the usual meaning of Feature that applies outside of CNN's, which is the aspects of your input data, e.g., for email classification, a feature could be the number of backlash characters in the Email text. This is related to the above since one can view a layer in a CNN as changing the ontology of the input space. (Instead of thinking in terms of "#words in email", "#backlashes in email", "word 'info' is in email yes/no", etc., it now thinks in terms of "concept of channel #1 in output layer #4", "concept of channel #2 in output layer 4", etc..) I assume this is why we call CNN-stuff "Features". But it's still clearly not the same.
- Neuron: According to this, I think it would be the third meaning of Feature I've listed, but is that really how most people use it?
