After pondering on the (skip-gram) word2vec algorithm and the fact that its single hidden layer is linearly activated, I am not 100% sure that I understand the significance of everything that is happening.

The skip-gram word2vec algorithm optimizes (Mikolov et al. (2013))

$$p(w_O | w_I) = \frac{exp({v'_{w_O}}^T v_{w_I})}{\sum_{w=1}^{W} exp({v'_w}^T v_{w_I})}$$ $v_w$ ($v'_w$) is the input (output) embeddings of the word $w_I$ (input word) and $w_O$ (context word) we are interested in. This algorithm is often represented as a neural network with an $n$-dimensional input layer that takes the sparse vector representing a word, an $m$-dimensional hidden layer (where $m < n$) with linear activation, and an $n$-dimensional softmax activated output layer.

The fact the word2vec uses linear activation on the hidden layer suggests that we can reduce the neural network to an $n$-dimensional input layer that connects directly to an $n$-dimensional softmax activated output layer (no hidden layer), and optimize a single square weight matrix ($V$).

If we take this approach of optimizing a single matrix, we could then perform some type of matrix factorization or decomposition, and decompose $V$ into 2 matrices of dimension $n \times m$ and $m \times n$. I originally thought that this way would allow us to test different embedding dimensions ($m$) while ensuring that the embeddings are optimal (given that $V$ is optimal) (no retraining needed).

However, it turns out that the math is not that simple. Such a decomposition of a square matrix ($V$) into 2 non-square matrices is only possible if $V$ is not full rank. Furthermore, if $rank(V)=r < n$, then it is only possible to decompose $V$ into an $n \times r$ and an $r \times n$ matrix.

Question: Could you please comment on the following conclusions:

  1. When we design our word2vec embedding, we can choose the embedding dimensionality. So, are we forcing $rank({{V'}_O}^T V_I) = m$ by training a word2vec with $m < n$ hidden nodes?
  2. By a training a network with one $n \times n$ weight matrix (i.e. no hidden layer) and calculating its rank, we could find the optimal dimension of the embedding layer (optimal in terms of lowest loss, not computational efficiency).

I hope I made my thoughts clear enough but would be happy to clarify.


I ran a small experiment and it seems to me that including the hidden layer does ensure a defined rank. I trained a model without the hidden layer and the resulting weight matrix was full rank.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.