0
$\begingroup$

Background: In word2vec we pass in a one-hot encoding of our target word into a simple neural network which is trained to predict context words from a window around our target. We eventually take the weights from our hidden layer to use as word embeddings - vector representations of our words.

When we put in a one-hot encoding into our hidden layer we end up essentially doing a look-up of the corresponding row in the weight matrix, so when training as finished we can say that this row represents the embedding for a particular word.

My question is: I am passing images through a pre-trained VGG network, and then use the final intermediary layer as my encodings for my images. I am then using these encodings as a replacement for my one-hot vectors and feeding them into a skip-gram architecture like W2V to learn an image embedding.

In this case the dot product of my input feature vector and the hidden layer weights are no longer a look-up affecting a single row - s how can I connect images with their embeddings in the this setup?

Edit: Some added context - I am trying to get style embeddings in a style vector space using this paper https://arxiv.org/pdf/1708.04014.pdf. The paper describes putting the images through a pre-trained VGG and then using these feature vectors to train the embedding layer. Normally we would then go and take the embedding layer weights, but in this case there isn't a single row that connects with our image because it isn't one-hot. Do we take the embedding layer activations instead of the actual weights?

$\endgroup$
0
$\begingroup$

I think one-hot encoding is just special representation of word. It is unnecessary. If you get a image word representation $v$ from last fc layer of VGG network, and $W$ is hidden layer of embedding, the dot product $v\cdot W$ will give you the embedding code.

$\endgroup$
0
$\begingroup$

I could be wrong, but from reading the paper it seems like the classification context was already known. Perhaps they did a look-up based on their already known bunches of style? (like the jeans could be 'casual' + 'bottoms' + 'black' or something like that, so the jeans are given the context of all casual, bottoms and black attire.)

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.