I was reading a paper that some Google researchers wrote (find it here: https://arxiv.org/abs/1603.02199) and having a look at their neural network, they have two images stacked on top of each other as one input and in the middle, they have a motor command (a series of scalars basically) tiled into the output of a max pool layer.

In their words: "The motor command is processed by one fully connected layer, which is then pointwise added to each point in the response map of pool2 by tiling the output over the special dimensions".

Can anybody explain how exactly this works? Or in other words, how would one insert a scalar or a series of scalars as an input to a CNN?


There are many ways to combine scalar and image inputs. In this particular paper, a diagram on the top of page 5 should explain everything. At some point in the convolutional network there are 64 feature maps, which matches the 64 scalar values to be input. The 64 scalar values are essentially treated as bias terms so that the $i$th scalar value is added on to the $i$th feature map.

Other popular methods of pulling this off usually inject scalar outputs after all the convolutional layers, when the last feature map has been flattened and the fully connected layers start, it is easy to concatenate in some auxiliary scalar inputs.

| cite | improve this answer | |
  • $\begingroup$ What does "pointwise added to each point in the response map by tiling the output over the special dimensions" mean though? (Page 5 Figure 4 Caption) $\endgroup$ – Saravanabalagi Ramachandran May 14 '18 at 15:02
  • 1
    $\begingroup$ $i$th scalar value is added to all values of $i$th feature map, isn't it? $\endgroup$ – Saravanabalagi Ramachandran May 14 '18 at 15:12

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.