Word embeddings - Pre-trained tokenizers vs more involved methods I'm drowning under all the various methods of converting my text corpora into embeddings.
I'm currently using the HuggingFace Tokenizer (https://github.com/huggingface/tokenizers) to do this, using the encode_plus method, which seems to do a great job of converting my text data into embeddings to be passed to my neural network.
However, I've noticed the vocabulary size of the tokenizer is about 30k, whereas word2vec vocab size is 3m. I have hardly any experience of using other methods than the HF Tokenizer, but there still seems to be lots of posts being written on blog sites about how to use W2V (etc). If the HF implementation is so great, why isn't everyone just using that? Is there any known advantage to obtaining embeddings the W2V way for a simple multiclass text classification problem?
Yes, HF is used primarily for converting to the very niche requirements of BERT (et al) inputs, but you can easily turn off the more complex outputs and just have a simple list of embeddings returned. I originally started out using BERT but found it too complex a model for my needs, so I just use the HF Tokenizer, and then a vanilla LSTM.
 A: HuggingFace's Tokenizers are just tokenizers, i.e., they do not make any embeddings. The encode_plus method only returns one-hot vector, so need to train embeddings on your own. (Either explicitly using an embeddings layer or implicitly in the first projection matrix of your model.)
Even with HuggingFace's Tokenizers, you can create a vocabulary as large as you want.
tokenizer = ByteLevelBPETokenizer()
tokenizer.train(["wiki.test.raw"], vocab_size=20000)

This snippet will create a byte-level BPE-based vocabulary with 2000 entries, however, you need to estimate the vocabulary parameters on some training data.
Machine translation and pre-trained contextual representations typically use vocabularies of tens of thousands of units, so that infrequent words get split into smaller units (using e.g., the WordPiece or the BPE algorithm) and the following layers will learn how to treat them together. This ensures that there are no out-of-vocabulary tokens and all vocabulary units get updated reasonably frequently during training.
On the other hand, wod2vec is able to learn embeddings for many words. The models trained with word2vec will probably generalize for words that are in the embeddings table but were never seen during the task-specific training. The disadvantage is that you cannot easily finetune the embeddings for your task, because it is very unlikely that your task-specific training data will contain all 3M unique word forms, so you can update them accordingly.
