I know there are approaches that process unknown words with their own embedding or process the unknown embedding with their own character neural model (e.g. char RNN or chat transformer).
However, what is a good rule of thumb for setting the actual min frequency value for when uncommon words are set to the unknown?
I am well aware that this is just a hyperparameters that could probably be chosen with the validation set (or even meta-learning or autoML methods) - but I really just want some heuristic to set that value. e.g. my expectation is that the bigger the model probably the larger the min-frequency has to be set to? Though I don't know exactly how to create this rule of thumb or if it exists. I assume this is a well known problem in NLP so I'd rather not make something up but base it on something that is more or less tested but extremely simple.
An answer that I expect would be some sort of expression or simple rule (or even a constant) I can use with little to no training. E.g. something simple like this:
min_freq = 0.5 * sqrt(number_parameters) + 0.5 * sqrt(vocab_size)
or perhaps (even better) a constant based on previous research papers for a value that tends to work with vocabs of certain size and models with certain number of parameters
min_freq = 30
e.g. I once heard that a good number of repetitions of an experiment is 30 based on the central limit theorem. A little made up but not totally random. The simplest but most validated method is probably the best.
I was also thinking something like this:
well there are 600 examples in mini-imagenet (has 64 labels). So a classifier probably needs around that number of examples for it be worth it. So let's make sure count() ~ 600 during training to have a meaningful representation for ukn.
Im currently going without something like that and printing this value:
print(torch.sum(torch.tensor(list(vocab.values())) < 30))
tensor(2605)
setting min_freq to a value large enough for unknown to get a meaninful value but that the other uncommon words also get enough gradient updates.
Nice related questions:
