I am using a set of pretrained word embeddings (GloVe) in my NN model. So far i just omit OOV words and i add zero-padding at the end of a sentence to have a fixed length for all the sentences/docs (for CNNs).
I want to add special padding tokens for the beggining/end of a sentence and for OOV words (
UNK respectively). My problem is what values should the embeddings of these tokens have. For the
UNK i was thinking to randomly initialize it from a uniform distribution (-0.5, 0.5).
np.random.uniform(low=-0.5, high=0.5, size=(10,)) array([ 0.48440237, 0.43741531, -0.46003191, 0.26630407, -0.26999162, 0.48189692, 0.16413969, -0.3330607 , 0.45768477, 0.14461057])
I am not even sure if that is correct. Also what abbout
I should note that my the embeddings are "frozen" / will not be trained any further for the task, so the values i will pick for the padding tokes will not change.
Is there a way to pick some sensible values?