1
$\begingroup$

I was reading this paper (https://papers.nips.cc/paper/5021-distributed-representations-of-words-and-phrases-and-their-compositionality.pdf) I cannot understand where does the multiplicative constant $1/T$ get from?

enter image description here

I understand that the objective function is to maximize the probability of any context words given the current center word:

enter image description here

When taking $\log$ there is no multiplicative constant. So, what is the clue?

Improve this question
$\endgroup$
2
  • $\begingroup$ Averaging the log likelihood tends to produce optima of around the same size when fitting models to datasets of substantially different sizes. The factor of $1/T$ ruins some powerful statistical interpretations coming from likelihood theory, though. $\endgroup$
    – whuber ♦
    Commented Feb 24, 2020 at 21:09
  • $\begingroup$ Does this answer your question? Motivation for average log-likelihood $\endgroup$
    – fnl
    Commented Feb 28, 2020 at 20:49

1 Answer 1

Reset to default
1
$\begingroup$

In theory, the multiplicative constant 1/T can be added ad-hoc to the objective function or not, it is irrelevant. Maximizing the objective function is equivalent to maximizing its Logarithm, and is also equivalent to maximizing its Logarithm times a constant.

For numerical optimization it can be an advantage to consider the average of the log-likelihood (by taking the average appears the 1/T factor you mention) instead of the unnormalized likelihood. The reason for this was explained here, and is related to having similar values of the log likelihood independently of the dataset size.

Improve this answer
$\endgroup$
2
  • $\begingroup$ But what is the intuition / meaning behind adding $1/T$? $\endgroup$
    – alienflow
    Commented Feb 24, 2020 at 15:21
  • $\begingroup$ Adding the 1/T is to take the average Log-Likelihood, I have edited the answer as I was omitting some important details $\endgroup$
    – Alejandro Celis
    Commented Feb 24, 2020 at 21:07

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.