# Could we drop the hidden layer in a skip-gram word2vec and train only a square weight matrix?

After pondering on the (skip-gram) word2vec algorithm and the fact that its single hidden layer is linearly activated, I am not 100% sure that I understand the significance of everything that is happening.

The skip-gram word2vec algorithm optimizes (Mikolov et al. (2013))

$$p(w_O | w_I) = \frac{exp({v'_{w_O}}^T v_{w_I})}{\sum_{w=1}^{W} exp({v'_w}^T v_{w_I})}$$ $$v_w$$ ($$v'_w$$) is the input (output) embeddings of the word $$w_I$$ (input word) and $$w_O$$ (context word) we are interested in. This algorithm is often represented as a neural network with an $$n$$-dimensional input layer that takes the sparse vector representing a word, an $$m$$-dimensional hidden layer (where $$m < n$$) with linear activation, and an $$n$$-dimensional softmax activated output layer.

The fact the word2vec uses linear activation on the hidden layer suggests that we can reduce the neural network to an $$n$$-dimensional input layer that connects directly to an $$n$$-dimensional softmax activated output layer (no hidden layer), and optimize a single square weight matrix ($$V$$).

If we take this approach of optimizing a single matrix, we could then perform some type of matrix factorization or decomposition, and decompose $$V$$ into 2 matrices of dimension $$n \times m$$ and $$m \times n$$. I originally thought that this way would allow us to test different embedding dimensions ($$m$$) while ensuring that the embeddings are optimal (given that $$V$$ is optimal) (no retraining needed).

However, it turns out that the math is not that simple. Such a decomposition of a square matrix ($$V$$) into 2 non-square matrices is only possible if $$V$$ is not full rank. Furthermore, if $$rank(V)=r < n$$, then it is only possible to decompose $$V$$ into an $$n \times r$$ and an $$r \times n$$ matrix.

Question: Could you please comment on the following conclusions:

1. When we design our word2vec embedding, we can choose the embedding dimensionality. So, are we forcing $$rank({{V'}_O}^T V_I) = m$$ by training a word2vec with $$m < n$$ hidden nodes?
2. By a training a network with one $$n \times n$$ weight matrix (i.e. no hidden layer) and calculating its rank, we could find the optimal dimension of the embedding layer (optimal in terms of lowest loss, not computational efficiency).

I hope I made my thoughts clear enough but would be happy to clarify.

EDIT:

I ran a small experiment and it seems to me that including the hidden layer does ensure a defined rank. I trained a model without the hidden layer and the resulting weight matrix was full rank.