Downweight or partially mask certain inputs to Neural Network I have an NLP classification task for sentences set up, in which the goal is to predict a sentence label that depends on the primary verb used in the sentence. This task can be solved by just memorizing the verb - label association, but I would like to regularize or "encourage" the model to use information from the surrounding context as well, so that the model can generalize well to unseen verbs. Fully masking the embedding corresponding to the verb results in an underspecified task, since information from the verb is needed to fully determine the label.
In short, I'd like a way to partially mask or downweight a specific input embedding to a neural network classifier, to encourage the network to use information from the surrounding context in addition to that input. I've thought about rescaling the verb embedding by a constant $c < 1$, but then $c$ becomes a hyperparameter that I would have to set somewhat arbitrarily.
Any suggestions or pointers to references would be greatly appreciated. Thanks!
 A: What you are asking is a really common problem in NLP, how do you generalise better to unseen words. You are talking about masking your primary verb so you can pick up on context around it more easily. Doing this may well work to some degree and dropout is a commonly used approach for neural networks generally.
But here is an alternative approach:
I'm not sure how you are representing your sentences, but I suspect it might be one hot encoding each word so that they are represented as [0,0,0,...0,1,0,...0] and your sentence is a sequence of these? This form could definitely leading to memorising of verbs as you said.
It is possible to transform into a vector space where synonyms of a words (primary verb or otherwise) lie very close to each other in the vector space. This could be done using the GloVe transform for example. For intuition on how this works, say the word 'were' never appeared in your dataset, but the word 'was' did. The GloVe transform was trained on a MASSIVE amount of data and so the common word 'were' has a place in the vector space, likely very close to the word 'were'. This then helps your NN to use what it learned about 'was' to help it use 'were' and so generalise better.
