# definition of "hidden unit" in a ConvNet

I'm 90% sure my definition is right, but it's such a core concept that I want to be sure.

When we're talking about a normal neural net, the definition of a hidden unit to me is clear.

But I learned about ConvNets from taking Andrew Ng's Deep Learning specialization, where in the context of ConvNets he normally talks about input/output volumes and filters. He never really defined a hidden unit in this context, so when he used the term I was confused.

Here's what I think the definition is. Can you tell me if I'm right?

A hidden unit corresponds to the output of a single filter at a single particular x/y offset in the input volume. So for example if your input volume is 9x9x3 and you have 5 3x3 filters (stride of 1 with no padding), your output will be 7x7x5, each filter is solely associated with 49 hidden units, each hidden unit is solely associated with one filter, and there are 49x5=245 hidden units at this layer.

Or I guess I could have said things more simply by saying that a hidden unit is the value at a particular x,y,z coordinate in the output volume.

Is all of that right? And just for the avoidance of doubt, a neuron still = a hidden unit here, right?

• Until 7*7*5 makes sense, Afterwards How you have arrived at the concept of the hidden unit, that doesn't make sense? In my opinion, you have (3*3*3) volumes that you will convolve(element-wise multiply & add) over your (9*9*3)input, 49 times for 1 filter since you have 5 of such kind, you will do the same convolve ops just 5 times more, therefore 49*5=245! Since in your CONV layer, a unit is acting as a single neuron is the sense A(W*X+b), it's just repeating the ops many times, wouldn't the number of the hidden unit be just 5 each of which is capacitated to use (f *f *n_c_prev) weights/vol?
– Anu
Commented Mar 23, 2019 at 22:59

Generally speaking, I think for conv layers we tend not to focus on the concept of 'hidden unit', but to get it out of the way, when I think 'hidden unit', I think of the concepts of 'hidden' and 'unit'. For me, 'hidden' means it's neither something in the input layer (the inputs to the network), or the output layer (the outputs from the network). A 'unit' to me is a single output from a single layer. So if you have a conv layer, and it's not the output layer of the network, and let's say it has 16 feature planes (otherwise known as 'channels'), and the kernel is 3 by 3; and the input images to that layer are 128x128, and the conv layer has padding so the output images are also 128x128. So, the outputs from that conv layer will be a cube of 32 planes times 128x128 images. To me, independent of the kernel size, there are 32x128x128 units in that layer's output.

However, typically, I think we tend to use language such as 'neurons' and 'units' for linear, otherwise known as fully-connected layers.

For conv layers, I feel that we specify them in terms of:

• feature planes, otherwise known as channels
• kernel size, eg 3 by 3
• padding, eg 1, at each edge
• stride, eg stride 1, in both directions
• (there's also some other stuff like dilation...)

And then we refer to things within this such as: - an output - an input - a 'feature plane' - a 'weight'

• Things aren't clear!.As per your answer input is (128*128*n_c_prev), CONV-layer has (3*3*n_c_prev) filter dimension with n_c=16 of such kind. Here, since you haven't defined n_c_prev, I took it as 1. Also, you are considering Padding=1 & stride=1("Same" convolution). Then the output dimension would be 128*128* n_c where n_c is 16. What is 16 feature planes come from and how you arrived at 32 in the output?. Also, filter dimension is always (ffn_c_prev) where n_c_prev depends on the number of the input channel, if you are using RGB image, it's 3(volume) if single channel then 1(plane)
– Anu
Commented Mar 23, 2019 at 22:35

I think @stephen & @hugh have made it over-complicated,

let's make it simple. A hidden unit, in general, has an operation Activation(W*X+b). Therefore, if you think carefully,

*A hidden unit in CONV layer is an operation that uses "filter_volume a.k.a volume of randomly initialized weights" in general. More loosely, you can say filter/filter volume (f *f n_c_prev) corresponds to single neuron/hidden unit in a CONV layer.

Particularly, in your example, you have (3*3* 3) filter volume that you will convolve (element-wise multiply & add--> Bias--> Activation) over your (9*9* 3) input.

(f* f* n_c_prev) is a filter in general, with n_c_prev as the number of the input channel,

49 (7*7) times for 1 filter, since you have 5 of such kind, you will do the same convolve operation just 4 times more,

therefore, 49*5=245 is the total number of convolution operation you are going to perform on the input using you 5 differently initialized filter volumes!

Therefore, the number of the hidden unit be just 5 each of which is capacitated to use (f *f *n_c_prev) weights/vol. Thinking more abstractly, a hidden unit in layer-1, will see only a relatively small portion of the neural network. And so if you visualize, if you plot what activated unit's activation, it makes sense to plot just small image patches, because that's all of the images that particular unit sees. Example

Now you pick a different hidden unit in layer-1 and do the same thing

Therefore, now you have 9 different representative neurons and each of them finds the nine(3*3) image patches that maximizes the unit's activation.

Now, if you deeper into the network, a hidden layer over there, a hidden unit sees a larger patch/region the image(larger receptive field!) and able to detect many complex patterns such as

More about it you can read here "visualizing and understanding convolutional networks"

I don't think either of the answers provides a clear definition, so I will attempt to answer it because I stumbled into the same problem finding a clear definition of a hidden unit in the context of a Convolutional Neural Network.

Hidden units in this context are the feature maps or filters. So for Tensorflow or Keras it would be

tf.nn.Conv2D(**hidden_units**,...)


Hidden Units based on the definition provided by http://www.cs.toronto.edu/~asamir/papers/icassp13_cnn.pdf

A typical convolutional network architecture is shown in Figure 1. In a fully-connected network like DNNs, each hidden activation hi is computed by multiplying the entire input V by weights W in that layer. However, in a CNN, each hidden activation is computed by multiplying a small local input (i.e. [v1, v2, v3]) against the weights W. The weights W are then shared across the entire input space, as indicated in the figure. After computing the hidden units, a maxpooling layer helps to remove variability in the hidden units (i.e. convolutional band activations)

There are so many complexities in this topic which may confuse one so let's break it down.

Here is what we have,

(Layerl1) (9 * 9 * 3) ---->Conv with(3 * 3 * 3),5 filters with s=1 p=0-----> (layerl2) (7 * 7 * 5)

now,

unit:- A units in a layer is that whoose receptive fields cover a patch of the previous layer

and,

1. no of hidden units in layerl2 = no of channels in layerl2

reason>

each filter detects a patch of region from previous layer layerl1 and each of this patch is called a unit of layerl2. and we know that no of channels in layerl2 = no of filters

1. units can share filters i.e. 2 patches can have same filter

reason>

let filter no 1 detects vertical edges, filter no 2 detects vertical bars, then if our image has some grid shape then both filter1 and filter2 will detect this grid, this grid is a patch which makes a unit, and this patch/unit is shared by 2 filters.

To Sum up:- Each channel of layerl2 is a hidden unit of layerl2 (here 5 hidden units )