How to extract vector representation from a comparison neural networks

Question

From what I understand about embeddings in neural networks, the upper layers (fully-connected) from convolutional neural networks can serve as an image encoding (vector representation of an image), and word2vec where the weights of a simple shallow neural net can provide a rich encoding of a word.

However, these are mostly single classification(cnn) or trying to maximize the likelihood(word2vec), now suppose the label does not depend on a single input but a pair, for a naive example, a pair input of an image of a square and an image of a triangle is given the label 0, and the pair of a line and a triangle is given the label 1, then after training a model that takes in a pair of images(I guess this can be done just by adding the second image as extra channels to the first image), how do I get a vector representation of the line, triangle and the square?

Or in other words, the rock, paper and scissors game, we can’t just classify one of the items to win or lose, they are intransitive. So we can still train a classifier to classify if the first person going to win with rock and the second person in the pair gives a paper, but how do we get some sort of embeddings for each of the rock, paper, and scissors?

These examples remind me of Siamese architecture in CNN, but the setup is for classifying if two inputs are same or not, I assume it doesn’t care about the order of the pair (e.g. the two different face are always different no matter the order in the pair, but the Rock Paper Scissors matters in order to who wins), plus the two subnets are sharing the same weights.

Any suggestions?

Answer 1

Embeddings in Neural Networks in general

You can often indeed interpret layers close to the output layer of a neural network as encoding an embedding / efficient representation of the inputs. This is not always true though. It's only true to the extent that (assuming successful training) it will encode parts of the input that are relevant for minimizing the loss function used during training.

For example, in those classification tasks with text or images that you mentioned, the contents of that text or image is likely to be highly relevant for the classification task, so in some sense you can expect to find those contents in a compact / efficient form (likely very difficult to interpret for a human) in the deeper layers of the network. If you were to use the same inputs for training, but completely random targets as output, you'd be less likely to find any meaningful embeddings after training.

---

Pairs of images as input

Now, let's consider your example with pairs of images as input:

now suppose the label does not depend on a single input but a pair, for a naive example, a pair input of an image of a square and an image of a triangle is given the label 0, and the pair of a line and a triangle is given the label 1, then after training a model that takes in a pair of images(I guess this can be done just by adding the second image as extra channels to the first image), how do I get a vector representation of the line, triangle and the square?

In your example, we have:

• Input: square + triangle. Output: 0.
• Input: line + triangle. Output: 1.

The contents of each of the images in the pair are still relevant for the predictions, so I'd still expect efficiently encoded representations of these images to be somewhere in the network. It's going to be very difficult, if possible at all, to tell where in the network it'll decide to encode what though. It may also just encode the complete input space (which is pairs of images) at once, it may not "cleanly" split up and have separate encodings of the separate images.

---

How to extract separate representations anyway?

If you really have to be able to extract separate representations for the images in your pairs of inputs, and you know this in advance, I think your best bet would be to modify your Neural Network's architecture and loss function used during training in advance. A neural network does not have to have just a single objective (like classification). You can split up into multiple different "output layers", each with their own loss function, and simply train using the gradients obtained by adding up all the loss functions (possibly a weighted sum of loss functions).

I'd recommend taking inspiration from auto-encoders, and adding additional outputs with additional loss functions in order to force your network to learn efficient representations of each of the images in the pair of images.

So, suppose your normal network would take two images as input, go through a few hidden layers, and perform classification in an output layer. From one of the hidden layers, you could additionally split up into two more "branches" of the network, which no longer connect to the original output layer, but connect into two additional output layers after a few more hidden layers. You could train these parts of the network like you would any other auto-encoder, train them to ultimately re-produce the pair of original input images in the two newly-added output layers. Then, the hidden layers in these "branches" of the neural network can be expected to each learn an efficient encoding of one of the original images, because you force this during training; these hidden layers eventually end up in output layers that must reproduce the original inputs, so they must in some way contain a representation of the inputs.