# Tag Info

## Hot answers tagged language-models

9

according to Dan Jurafsky and James H. Martin book: "It turns out, however, that simple frequency isn’t the best measure of association between words. One problem is that raw frequency is very skewed and not very discriminative. If we want to know what kinds of contexts are shared by apricot and pineapple but not by digital and information, we’re not going ...

6

Figure 1 there clarifies things a bit. All word vectors from window of a given size are summed up, result is multiplied by (1/window size) and then fed into output layer. Projection matrix means a whole lookup table where each word corresponds to single real-valued vector. Projection layer is effectivly a process that takes a word (word index) and returns ...

6

Dan Jurafsky has published a chapter on N-Gram models which talks a bit about this problem: At the termination of the recursion, unigrams are interpolated with the uniform distribution: \begin{align} P_{KN}(w) = \frac{\max(c_{KN}(w)-d,0)}{\sum_{w'}c_{KN}(w')}+\lambda(\epsilon)\frac{1}{|V|} \end{align} If we want to include an unknown word &...

5

As you noticed, it's good idea to have some kind of averaging. Since in LM probabilities get multiplied, geometric average seems like a good fit. From Speech and Language Processing In practice we don’t use raw probability as our metric for evaluating language models, but a variant called perplexity. The perplexity (sometimes called PP for short) of a ...

5

The main advantages of using strict probabilities are a) ease of interpretation of the numbers; and b) being able to use Bayes theorem and other probabilistic methods in subsequent analysis. In some situations though, it is unnecessary. For example if you just want to rank the results with no further analysis, then there's no need to normalise the scores.

4

Michel, Jean-Baptiste, et al. "Quantitative analysis of culture using millions of digitized books." science 331.6014 (2011): 176-182. is the publication that describes the data set: The resulting corpus contains over 500 billion words, in English (361 billion), French (45B), Spanish (45B), German (37B), Chinese (13B), Russian (35B), and Hebrew (2B). The ...

4

Yes. That will generate many more features though: it might be important to apply some cut-off (for instance discard features such bi-grams or words that occur less than 5 times in your dataset) so as to not drown your classifier with too many noisy features.

3

The skipgram model produces one softmax output for all the context words. The drawing from the lecture is incorrect.

3

CLD2 uses sequences of four letters (quadgrams, or Ngrams with N=4) for most Unicode scripts. The training text came from millions of web pages, filtered by several years' worth of off-and-on software development to use previous versions and high thresholds to remove suspect text, then cross-checked against extensive test data. Plus a lot of manual reading ...

3

The task of finding missing words in a text sometimes referred to as text imputation, or sentence completion. One paper exploring it with ANN: Solving Text Imputation Using Recurrent Neural Networks. Arathi Mani. CS224D report. 2016. http://cs224d.stanford.edu/reports/ManiArathi.pdf In this paper, we have shown that the bidirectional RNN yields the best ...

3

The number of bigrams can be reduced by selecting only those with positive mutual information. We did this for generating a bag of bigrams representation at the INEX XML Mining track, http://www.inex.otago.ac.nz/tracks/wiki-mine/wiki-mine.asp. What we did not try is using the mutual information between the terms in weighting the bi-grams. See https://en....

2

Multiplying a vector with a matrix A followed by another matrix B can be written as multiplication by an equivalent single matrix. However, the number of elements in the matrix AB can be much higher than number of elements in the matrix A + number of elements in the matrix B. Example in CBOW: $A = W$, $B = W'$. The number of elements in the matrix AB is $N^... 2 The main advantage of working with character-level generative models is that the discrete space you're working with is much smaller -- there are about 97 English-language characters in common usage if we include all punctuation marks. By contrast, a vocabulary is many thousands of words. This implies that just storing the word embeddings will require a lot ... 2 It seems you have misunderstood how W2V algorithms work. Both W2V algorithms (Skip-Gram, Continuous BoW) use dense vectors initialized randomly which are optimized afterwards. For skip-gram, because of this dense representation, context words are predicted in a sequence. (Christopher Moody does a great job explaining Skip-Gram here) If you're familiar ... 2 Neural network language models usually take their input in the form of one-hot encoded vectors. This means that the dimensionality of the vector is equal to the size of the vocabulary and that the vector is all zeros except for a single one at the position that corresponds to the id number for that word. The one-hot input vectors are multiplied by an matrix ... 2 The estimate$100 000^{10}-1$comes from assuming a discrete model for the$10$consecutive words, without any simplifications or restrictions, thus using all interactions up to and including order$10$. It is not important that the words are consecutive, we would get the same count for any ten specified word positions. For each position, it can be any of ... 2 Edit: My answer is wrong, read the answer below from Dick Sites, the main author of CLD2. My guess: There is a concept of priors: the probability that text generally is, say, Danish. As we know, the real datasets and real queries are very imbalanced, they contain much more English than other languages. Concrete example: 50% of the actual text is ... 2 You can solve this problem by fitting only the more complex model, then transforming the resulting parameter estimates$q^*$to recover$q$and$r$. No iterative optimization needed! Please forgjve me if jn thjs answer J have mjxed up$j$and$i$. Jt's iust very djffjcult for me to keep them strajght. Suppose your dataset consists of one observation of ... 2 Aside from unsupervised methods like doc2vec, there are couple of supervised methods: Siamese network: github example, What are Siamese neural networks DSSM StarSpace All of them aims to create vector representations for documents, so dot product of vectors would represent semantically similar of documents. 2 According to the paper, the n-gram model predicts the next word based on the previous n words. Let's get intuitive... Say n is 4, and you have "I love to eat". That is, given "I love to eat ______", you want to fill in the blank with the most likely word. Intuitively, I would guess that "I love to eat ____" and "I like to eat ____" would have a similar ... 1 In the topic models category there is also NMF (Non-negative matrix factorization) 1$N$in the equation corresponds to the N in N-gram. E.g. if you use bi-grams,$N=2$. 1 Negative log-likelihood and negative likelihood both have minima in the same location because the logarithm is a monotonic injective transformation. In terms of optimization, they are the same. In terms of numerics, one generally prefers to avoid exponentiation since this can cause overflow errors. Note that we can use algebra to re-arrange from one ... 1 Choosing appropriate statistic for getting point estimate from posterior distribution is like choosing appropriate statistic for describing your sample. We use mean when we want our estimator to be sensitive to outliers, what leads to solution that is in the middle of the mass of the distribution, i.e. we are interested in the expected value. Median is the ... 1 Generally the hidden states are the parts of speech (eg, noun, verb) and the observations are the words. So we assume that each word (emission) depends only on the part of speech and each part of speech depends only on the part of speech preceding it in the sequence (this last one is "markov", or memoryless, assumption). The part of speech is therefore "... 1 Neural language model or another model takes numbers as input and they do not process/work directly over the string (or text ) so you need to convert them into the required format. One common approach is using bag of words model that you can create by counting the number of times a word is coming into the sentence and making a one hot vector with dimension ... 1 EDIT Feb 2, 2019: I figured I made a mistake below with the continuation count parameter (and with how a wolf sounds). I corrected it but published a new version in this link. If you might find it useful, please follow it there. I got here willing to find an answer but got no luck :( But then, I had to figure things out on my own! So let me try to explain ... 1 It means that in order to compute the$k$-gram probability distribution, you first need to compute the$(k-1)$-gram probability distribution. 1 Short answer: although it's possible to use it in this strange way, Kneyser-Ney is not designed for smoothing unigrams, because in this case its nothing but additive smoothing:$p_{abs}\left ( w_{i} \right )=\frac{max\left ( c\left ( w_{i} \right )-\delta ,0 \right )}{\sum_{w'}^{ }c(w')}$. This looks similar to Laplace smoothing and it is very well-known ... 1 Language models are often used to compute the probability of a sentence. This is done by using the chain rule. For example if we want to estimate the probability of observing the sentence$w_1 w_2 w_3 w_4$we can factorize it like so...$P(w_1, w_2, w_3, w_4) = P(w_4|w_3, w_2, w_1) P(w_3|w_2, w_1) P(w_2| w_1) P(w_1) \$ Each of those terms is something that ...

Only top voted, non community-wiki answers of a minimum length are eligible