1
$\begingroup$

After pondering on the (skip-gram) word2vec algorithm and the fact that its single hidden layer is linearly activated, I am not 100% sure that I understand the significance of everything that is happening.

The skip-gram word2vec algorithm optimizes (Mikolov et al. (2013))

$$p(w_O | w_I) = \frac{exp({v'_{w_O}}^T v_{w_I})}{\sum_{w=1}^{W} exp({v'_w}^T v_{w_I})}$$ $v_w$ ($v'_w$) is the input (output) embeddings of the word $w_I$ (input word) and $w_O$ (context word) we are interested in. This algorithm is often represented as a neural network with an $n$-dimensional input layer that takes the sparse vector representing a word, an $m$-dimensional hidden layer (where $m < n$) with linear activation, and an $n$-dimensional softmax activated output layer.

The fact the word2vec uses linear activation on the hidden layer suggests that we can reduce the neural network to an $n$-dimensional input layer that connects directly to an $n$-dimensional softmax activated output layer (no hidden layer), and optimize a single square weight matrix ($V$).

If we take this approach of optimizing a single matrix, we could then perform some type of matrix factorization or decomposition, and decompose $V$ into 2 matrices of dimension $n \times m$ and $m \times n$. I originally thought that this way would allow us to test different embedding dimensions ($m$) while ensuring that the embeddings are optimal (given that $V$ is optimal) (no retraining needed).

However, it turns out that the math is not that simple. Such a decomposition of a square matrix ($V$) into 2 non-square matrices is only possible if $V$ is not full rank. Furthermore, if $rank(V)=r < n$, then it is only possible to decompose $V$ into an $n \times r$ and an $r \times n$ matrix.

Question: Could you please comment on the following conclusions:

  1. When we design our word2vec embedding, we can choose the embedding dimensionality. So, are we forcing $rank({{V'}_O}^T V_I) = m$ by training a word2vec with $m < n$ hidden nodes?
  2. By a training a network with one $n \times n$ weight matrix (i.e. no hidden layer) and calculating its rank, we could find the optimal dimension of the embedding layer (optimal in terms of lowest loss, not computational efficiency).

I hope I made my thoughts clear enough but would be happy to clarify.


EDIT:

I ran a small experiment and it seems to me that including the hidden layer does ensure a defined rank. I trained a model without the hidden layer and the resulting weight matrix was full rank.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.