Combining image and scalar inputs into a neural network

Question

I was reading a paper that some Google researchers wrote (find it here: https://arxiv.org/abs/1603.02199) and having a look at their neural network, they have two images stacked on top of each other as one input and in the middle, they have a motor command (a series of scalars basically) tiled into the output of a max pool layer.

In their words: "The motor command is processed by one fully connected layer, which is then pointwise added to each point in the response map of pool2 by tiling the output over the special dimensions".

Can anybody explain how exactly this works? Or in other words, how would one insert a scalar or a series of scalars as an input to a CNN?

Answer 1

There are many ways to combine scalar and image inputs. In this particular paper, a diagram on the top of page 5 should explain everything. At some point in the convolutional network there are 64 feature maps, which matches the 64 scalar values to be input. The 64 scalar values are essentially treated as bias terms so that the $i$th scalar value is added on to the $i$th feature map.

Other popular methods of pulling this off usually inject scalar outputs after all the convolutional layers, when the last feature map has been flattened and the fully connected layers start, it is easy to concatenate in some auxiliary scalar inputs.

Answer 2

The features vector can be combined to an image by -

1. Adjusting the features shape by using tf.reshape and tf.tile
2. Combining the features and image by performing concatenation, add (as described in Research document) or other merge operators

Here is a code example for creating a Custom Keras Layer that merge features and image, by using tile and concatenation -

class FeatureConcatLayer(tf.keras.layers.Layer):
  def build(self, input_shape):
    self.image_shape = input_shape[0][1:]
    self.num_features = input_shape[1][1]
    
  def call(self, inputs):
    image, features = inputs
    features = tf.reshape(features, (-1, 1, 1, self.num_features))
    features = tf.tile(features, [1, self.image_shape[0], self.image_shape[1], 1])
    return tf.concat([image, features], axis=-1)