I have a paragraph and I want to get the probability (p(word | context) ) of each word, given previous words, for various models (e.g. pre-trained LSTM).

Where can pretrained models would allow me to do this? Can someone provide an example as to how to fo this in Python?

  • $\begingroup$ What do you mean by 'predictability'? $\endgroup$ – jkm Dec 25 '19 at 9:56
  • $\begingroup$ @jkm sorry - p(word | context) $\endgroup$ – okuoub Dec 25 '19 at 10:54

Without having a model trained for this specific task, I don't think this is possible. However, if you allow the following assumptions to be true, one approach would be as discussed below.


  • The tokenizer of the pre-trained model supports your new corpus
  • You are allowed (re-)train/fine-tune using a subset of your corpus

Approach Summary

After splitting your corpus into train/validation, create the following using your train dataset - for every sentence in the train corpus, randomly drop tokens with probability p and build a classifier to predict the dropped token. Use validation corpus to pick the best model like usual.

The classifier can be another neural network attached to whatever pre-trained network which acts like a black-box embedding layer.


BERT (https://arxiv.org/abs/1810.04805) and variants explores this idea in detail to build Masked LMs and have at least performed very well on benchmarks.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.