Skip to main content
edited title
Link
Antoine
  • 6.2k
  • 8
  • 34
  • 56

Language models and curse Curse of dimensionality with language models

edited tags
Link
Antoine
  • 6.2k
  • 8
  • 34
  • 56
added 24 characters in body
Source Link
Antoine
  • 6.2k
  • 8
  • 34
  • 56

In the seminal paper A Neural Probabilistic Language Model, Yoshua Bengio and his colleagues make the following point:

If one wants to model the joint probability distribution of 10 consecutive words in a natural language with a vocabulary $V$ of size $100,000$, there are potentially $100,000^{10}-1$ free parameters.

I guess it's related to degrees of freedom and joint distributions but I just can't get my hands on the exact formula that was used here to come up with $100,000^{10}-1$.

In the seminal paper A Neural Probabilistic Language Model, Yoshua Bengio and his colleagues make the following point:

If one wants to model the joint probability distribution of 10 consecutive words in a natural language with a vocabulary $V$ of size $100,000$, there are potentially $100,000^{10}-1$ free parameters.

I guess it's related to degrees of freedom but I just can't get my hands on the exact formula that was used here to come up with $100,000^{10}-1$.

In the seminal paper A Neural Probabilistic Language Model, Yoshua Bengio and his colleagues make the following point:

If one wants to model the joint probability distribution of 10 consecutive words in a natural language with a vocabulary $V$ of size $100,000$, there are potentially $100,000^{10}-1$ free parameters.

I guess it's related to degrees of freedom and joint distributions but I just can't get my hands on the exact formula that was used here to come up with $100,000^{10}-1$.

Source Link
Antoine
  • 6.2k
  • 8
  • 34
  • 56
Loading