What will be the best way to estimate the entropy/surprise of a word in a specific context? Let's say to compare the surprise of: context:

"I watched the movie in my"



I thought about:

entropy(word | context) = entropy(I watched the movie in my computer) - entropy( I watched the movie in my)

What do you think? is this a good method? Are there better ones?

  • $\begingroup$ What I immediately thought of is wrongly spelled words are also very surprising $\endgroup$ – Minh Triet Sep 12 '18 at 8:38
  • $\begingroup$ @MinhTriet You are correct but I use a validated corpus so there are no mis-spelling $\endgroup$ – okuoub Sep 12 '18 at 8:59
  • $\begingroup$ Karpathy used RNN to predict next word in a sentence here. He calculates the next word given the previous words, should give you some hints $\endgroup$ – Minh Triet Sep 13 '18 at 8:33
  • $\begingroup$ @MinhTriet would you say RNN in better for this task then "regular" n-gram model? $\endgroup$ – okuoub Sep 17 '18 at 9:25
  • $\begingroup$ Size of n-gram models heavily depends on how large your corpus is, and RNN does not have that limitation. I am under an impression that RNN can be faster than n-grams, because you do not have to search through a huge probability dictionary $\endgroup$ – Minh Triet Sep 21 '18 at 21:43

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.