How is an embedding space optimized with respect to the loss function? I understand that the point of the embedding layer is to reduce the dimensionality of the input space while also projecting it onto a space that represents the similarity between the medium in question (eg. an image, a word, n-gram, etc).
What I can't understand is how the embedding vector for each point in the training space is updated. I understand that the embedding vector would start as a random vector. But how does optimization (eg. gradient descent, ADAM) update these vectors? Is the gradient applied pointwise like we do when updating neural network weights? So like this:
$$W = W + dW$$
, or is there some other method?
 A: If we create a very simple NN with only one Embedding Layer in the model, it could be seen more clearly. For example, in the model:
Input: [batch_num, 2] initialized to be [[0, 1]] (batch_num to be 1)
Embedding Layer shape: [3,2] with weights initialized to be:
[[0, 1],
 [2, 3],
 [4, 5]]

Output: [batch_num, 2, 2]
If we feed the ground truth label with shape: [1, 2, 2] (batch_num=1), for example, initiating it to be:
$\bar{y}$=
[
[1, 1],
[1, 2],
]

Apparently the output would be the selected 1st row and the 2nd row of the Embedding layer weights (or look-up-table):
y =[
[0, 1],
[2, 3],
]

Assuming we use simple loss function: MeanAbsoluteError()
The loss $L = \bar{y} - y$, which is just the difference of the two matrices. Note that the output y is just the identity of the Embedding Layer's weights w. To calculate the derivative of the w is essentially calculate the
$\frac{dL}{dy}$or$\frac{dL}{dw}$ (they are identities due to it's Embedding Layer). And the update process is: $w = w - \frac{dL}{dw}$ (assuming learning_rate is 1). For individual $w_i$, the update process is:
$w _{i}= w_{i} - \frac{d(\frac{\sum_{i=1}^{4} |y_{i} - w_{i}|}{4})}{dw_{i}}$
For example: $w_4 = w_4 - \frac{3 - 2}{4}$
I think the take away is that: the weights in the Embedding Layer got updated to reduce the loss. Whatever calculation immediately following the Embedding Layer only uses its weights as itself, thus, to calculate the gradient during the backward process, the derivative is done as usual.
