How to handle big vocabulary size with keras tokenizer? I am actually working on a neural language model developed with keras. 
I have an encoder and a decoder and the output of the decoder is a dense vector on the vocabulary..so quite big depending on the vocabulary. It's like a multi-class classification problem where each class is a word.
I have 440K unique words in my data and I use the tokenizer provided by Keras. So when creating my target vector data with np.zeros() (sparse matrix, one hot encoded) that is (length of sequence * number of sequence * size of vocabulary), I am facing a memory problem.
Do you how can I handle this ? Thank you 


*

*Reduce size of the training batches

*Reduce size of vocab

*Change the output shape

*Try character level language model

*Other idea

 A: 
Reduce size of the training batches

This could work if the main part of your memory consumption is the training batches. I'm not sure that is the case for you, though. As far as I can tell, the GloVe vectors will consume a few GBs on their own and then all the remaining parameters of the network could easily be just as large.

Reduce size of vocab

Are all of the GloVe words in your corpus? That's an obvious place that you could make some cuts, with no cost to your model. Another option is to cut the words that appear least-often in your corpus, since words with 1 or 2 appearances won't make much difference, but could go a long way to truncate the "long tail" of rare words.

Change the output shape

I think this is a promising option. You write "So when creating my target vector data with np.zeros() (sparse matrix, one hot encoded) that is (length of sequence * number of sequence * size of vocabulary), I am facing a memory problem"  -- Why are you doing it this way? Representing the target as an index would make the target much smaller, because the last axis wouldn't have extent size_of_vocabulary. The way this works hinges on two facts: (1) the target vector is 1-hot (2) we know that the value of the positive element must be 1; therefore, it's entirely unnecessary to store all of those 0 values because we only need to record which index is 1. So when you're computing the loss, this strategy means you only have to pay attention to the single element of the prediction vector which is indexed by the integer-encoded target.
To understand this in more detail, consider the log loss with a vocabulary of size $N$. If $y$ is one-hot, then only element $j$ is nonzero: $y_j=1$. This simplifies the expression of the loss dramatically.
$$
\begin{align}
\mathcal{L} &= \sum_{i=1}^N y_i \log\left( \hat{y}_i\right) \\
&= \sum_{i=1}^N \mathbb{I}(i = j) \log\left( \hat{y}_i\right) \\
&= 0 + 0 + \cdots + 1 \cdot \log\left(\hat{y}_j\right) + \cdots + 0
\end{align}
$$
This should make it clear that the integer coding strategy just stores the value of $j$.
For example, "The quick brown fox" could be coded as [1,2,3,4] if "the" is coded as 1 and "quick" is coded as 2 and "brown" is coded as 3 and "fox" is coded as 4. This scheme is more efficient than the one-hot encoding in terms of memory usage, since you don't have to store all of those zeroes value. If you used this encoding with 1-hot vectors, you'd have to store a matrix
$$
\begin{bmatrix}
1 & 0 & 0 & 0 & 0 ~\cdots\\
0 & 1 & 0 & 0 & 0 ~\cdots \\
0 & 0 & 1 & 0 & 0 ~\cdots\\
0 & 0 & 0 & 1 & 0 ~\cdots\\
\end{bmatrix}
$$
If you prefer to look at code, then https://www.tensorflow.org/api_docs/python/tf/nn/sparse_softmax_cross_entropy_with_logits provides an example of what I'm talking about.

Try character level language model

Character-level models can work really well for generative tasks, among others. I'm not sure how well a character-level model would work for your encoder-decoder task; I think you'll have to do an experiment to assess whether it would work for your problem.
