I look at Keras's document for Embedding layer and it doesn't say which method, skip-gram or negative-sampling, is used for training. I can't find any information online either.
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
2
The Embedding layer is basically just a table that stores the embeddings. It can look up embeddings for provided indices and compute the gradients in the back-ward pass. The layer does not care where the gradients come from.
You indeed can implement the word2vec algorithms in Keras, but this not how the Embedding layer is typically used. It is typically part of a more complex model (e.g., used as an input into an RNN) that is trained for some task (e.g., classification), and training signal comes from that task. It gets back-propagated through the entire network up to the embeddings, which is how the embeddings get the gradient to be updated with.
-
$\begingroup$ If it is just a table, then an embedding layer is no different from a dense layer, correct? $\endgroup$– etangOct 6, 2020 at 21:45
-
2$\begingroup$ The embedding layer picks up vectors from a weight matrix based on the indices. The dense layer does matrix multiplication and adds bias. You can indeed implement embeddings using a dense layer with one-hot vectors as an input, but it would waste precious GPU memory. $\endgroup$– JindřichOct 6, 2020 at 22:38