Stack Exchange Network

Stack Exchange network consists of 175 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange
The results are in! See what nearly 90,000 developers picked as their most loved, dreaded, and desired coding languages and more in the 2019 Developer Survey.

Questions tagged [language-models]

A statistical language model is a probability distribution over sequences of words.

0
votes
0answers
17 views

Good turing smoothing for unigram LM

I was wondering if it is at all possible to use good turing smoothing for unigram language model? I know that this smoothing technique helps distribute the weights from most occurring words to less ...
0
votes
1answer
24 views

How to train a RNN language model?

I want to train a RNN-based language model from https://arxiv.org/pdf/1409.2329.pdf for next word prediction. How to split the sentences from the dataset into input and ground truth during the ...
0
votes
1answer
28 views

Difference between ELMo and a normal deep network

I was reading up about ELMo and what I could gather was that we essentially combine the weights from different stacked lstm/gru layers for a given token as different layers are suited for different ...
0
votes
1answer
21 views

Likelihood for a test data (sequence of characters) given two unigram models

I would like to find the likelihood of a sequence of characters (the test data), given two unigram models. The sequence (test data) is: A B C B B The models ...
0
votes
1answer
18 views

Modeling words in a language based on their characters

I have different sets of strings, where I assume that each set follows some rules or patterns. For example, the first character must be a number, or the 3rd and the last characters must be the same, ...
0
votes
1answer
18 views

CMUDict normalized for word frequency

I am trying to train a neural grapheme to phoneme (G2P) model on CMUDict, but I find that pretty soon its loss is barely decreasing. Also, when I train the model on a different similar-sized dataset (...
0
votes
0answers
54 views

How to prove that katz backing off smoothing technique is a valid probability distribution?

How to prove that katz backing off smoothing technique is a valid probability distribution? Take the example of bigram. You can go through this link for katz back off model https://en.wikipedia.org/...
0
votes
0answers
51 views

How to sample a language model?

I've successfully trained a language model using LSTMs. But I have a confusion about sampling. On sampling, we generate a probability distribution at each time step. It will be of length vocabulary ...
0
votes
1answer
246 views

Where can I find pre-trained language models in English and German?

Where can I find (more) pre-trained language models? I am especially interested in neural network based models for English and German. I am aware only of Language Model on One Billion Word Benchmark ...
0
votes
1answer
159 views

Perplexity of a Non-Statistical Language Model

I have a piece of software that, given a input phrase, returns an ordered list of the next most likely words (entire vocab is ordered 1 to n). This is essentially an Language Model with the exception ...
2
votes
1answer
739 views

How to handle big vocabulary size with keras tokenizer?

I am actually working on a neural language model developed with keras. I have an encoder and a decoder and the output of the decoder is a dense vector on the vocabulary..so quite big depending on the ...
1
vote
1answer
45 views

How does clustering improve a language model?

This article describes a hierarchical clustering algorithm which clusters the words within a vocabulary based on their similarity, in order to improve a language model (in the article, n-grams). How ...
4
votes
0answers
34 views

natural language understanding algorithms [closed]

I am doing some research into how smart personal assistants work like siri, alexa etc. I have found that it is using automatic speech recognition to turn the speech into weighted text form and then ...
0
votes
0answers
89 views

Normalize probability distribution by variance of each class

I have several topics that I will collect language data for. Using Mturk I will ask responders to write sentences for each topic. The sentences will be used to train a language model. Language models ...
1
vote
2answers
53 views

What are commonly used methods to represent a document by a vector?

Methods that I know of Bag of words + weighting: tf-idf, bm25 Topic models: LSA, LDA Word/sentence/document embedding Are there other commonly used methods to represent a document by a vector?
1
vote
1answer
78 views

How to compute context-independent token representations in a biLM?

I've been reading this paper on ELMo word representations. For context, here's my understanding of the standard bi-directional language model (biLM) thus far: Given a sequence of tokens $(t_{1}, ...
1
vote
1answer
22 views

What the 'N' means on the following N-gram approximation

What the 'N'(capital) means in the following N-gram approximation to the conditional probability of the next word in a sequence ?
1
vote
0answers
20 views

Character level RNN for converting word forms

I want to build a char RNN to convert word form from one to another, for example, singular nouns such as lion to lions. However ...
0
votes
1answer
369 views

Perplexity calculation with neural nets

I am having troubles understanding which formula to use to calculate perplexity of a neural language model. Various places online on the forums people suggest using 2^(cross-entropy) measure, which is ...
1
vote
1answer
590 views

Skip-gram algorithm confusion

As a newbie to NLP, I am (deeply) confused by the middle step in the following diagram explaining the skip-gram algorithm. The video where this diagram was presented can be found at: https://www....
2
votes
1answer
108 views

How do we pass data to a RNN?

Let's say we have A1, A2, ... , Am different articles in the corpus and each of them has W1, W2, ....., Ww words. We are training a language model on them. Do we: Scheme 1 Take the first batch of ...
1
vote
1answer
69 views

Finding a mixture of 1st and 0'th order Markov models that is closest to an empirical distribution

I am interested in finding the distribution "$p^*$" closest to an empirical distribution $\hat{p}$ where $p^*$ is a mixture of first and zeroth order Markov models. That is, I want to find $$ p^* = \...
0
votes
1answer
317 views

Bayesian smoothing using Dirichlet prior : why not MAP?

I am reading about smoothing methods for language model ( I am working on unigram model). If you are not familiar with unigram model, it is closely related to multinomial distribution (with the ...
0
votes
0answers
35 views

Decide threshold for LM to filter out ill-formed sentence

I have a corpus of sentences. This corpus contains well-formed as well as ill-formed sentences. I want to filter out ill-formed sentences from the corpus. Ill-formed sentences are of two types : ...
2
votes
1answer
748 views

Language Model compare probability scores between Length varying sentence

My question is : How can I compare Language Model(LM) score for two sentences with different lengths ? Probabilities are < 1...
1
vote
1answer
617 views

what are hidden states in HMM based language model?

There are several ways to build language models, n-gram based models are straightforward, but for the language models built on HMMs, what are hidden states and what are observations?
1
vote
1answer
127 views

Language model created with SRILM does not sum to 1

I created an n-gram language model on the Penn Treebank using the following command: ...
0
votes
0answers
49 views

Classifying Sentences when outcome is a sentence

I am completely new to text classification and facing the following problem, I want to classify sentences and have a labeled dataset, the thing is that my Y variable is itself a sentence, hence I ...
3
votes
2answers
351 views

Language Modelling using Neural Networks

I plan to make a Language Model in Python using Neural Networks. I've read that Neural Networks need vectors as input. One common vector representation in NLP is the Bag of Words model. Given a corpus ...
1
vote
0answers
233 views

Deciding on the training sequences for RNN/LSTM language model

In a character language model, text is seen as a stream of characters. Say we have a training text as a string s, with length ...
2
votes
0answers
721 views

Computing Training Set Perplexity of a Neural Language Model: Too low values

I am implementing a Language Model based on a Deep Learning architecture (RNN+Softmax). The cost function I am using is the cross-entropy between the vector of probabilities at the softmax layer and ...
2
votes
2answers
651 views

Language Detection with CLD2 with Mixed Inputs in long documents

Internals Recap. CLD2 is a Naïve Bayesian classifier, trained on documents of mean size of 200 characters, trained on a corpus of 100M scraped and human expert selected web pages. When working on ...
6
votes
2answers
1k views

Calculating test-time perplexity for seq2seq (RNN) language models

To compute the perplexity of a language model (LM) on a test sentence $s=w_1,\dots,w_n$ we need to compute all next-word predictions $P(w_1), P(w_2|w_1),\dots,P(w_n|w_1,\dots,w_{n-1})$. My question ...
1
vote
1answer
960 views

A simple numerical example for Kneser-Ney Smoothing

I'm working in a project trying to implement the Kneser-Key algorithm. I think I got up to the step of implementing this formula for bigrams: $P_{(KN)}(w_i|w_{i-1}) = \frac{max(c(w_{-1}, w_{1}) - \...
0
votes
1answer
3k views

How to calculate the perplexity of test data versus language models

I have been working on an assignment where I train upon 3 corpora in 3 separate languages, and then I read in a set of sentences and use a number of models to determine the most likely language for ...
1
vote
0answers
348 views

Feature extraction from strings

We are given a big collection of strings, and an intensity associated with each string in the collection. In a sense, a 'distribution' on the dictionary. We are given that the intensity of each string ...
3
votes
1answer
2k views

Understanding Add-1/Laplace smoothing with bigrams

I am working through an example of Add-1 smoothing in the context of NLP Say that there is the following corpus (start and end tokens included) ...
1
vote
1answer
90 views

Under periodic BPTT, is softmax evaluated only at the end of the period?

Suppose I have a continuous sequence $X$ of words and I wish to train a RNN language model. According to [1], I would split $X$ into subsequences $X^{1..|X|/k_1}$ $k_1$ sized subsequences ($k_1$ is ...
6
votes
2answers
1k views

Neural network language model - prediction for the word at the center or the right of context words

Neural network language model - prediction for the word at the center or the right of context words? On Bengio's paper, the model predicts probability by n words for the next word, like predicting ...
2
votes
1answer
1k views

In Kneser Ney smoothing, how to implement the recursion in the formula?

I'm working in a project trying to implement the Kneser-Key algorithm. I think I got up to the step of implementing this formula for bigrams: $P_{(KN)}(w_i|w_{i-1}) = \frac{max(c(w_{-1}, w_{1}) - \...
4
votes
1answer
2k views

Advantage of character based language models over word based

Is there an intuition why character based models language bases models are preferred over word based. For example Karpathy builds his language model by predicting the next character in Karpathy Blog. ...
0
votes
1answer
482 views

Does the skipgram language model try to predict all context words at the same time?

In the skipgram language model (Mikolov et al., 2013), a neural network with one hidden layer tries to predict surrounding words from current words of the corpus. After training, the hidden activation ...
0
votes
2answers
768 views

Kneser-Ney for unigrams?

I was wondering if it is at all possible to use Kneser-Ney to smooth word unigram probabilites? The basic idea behind back-off is to use (n-1)-gram frequencies when an n-gram has 0 count. This is ...
1
vote
0answers
1k views

Katz Backoff help calculating alpha [closed]

The trigram version of Katz backoff is represented as follows (we refer to three words in a sequence as $x$, $y$, $z$ in that order): $P_{katz}(z|x,y) = \begin{cases} P^*(z|x,y) & \quad ...
2
votes
0answers
201 views

Alpha on Katz Backoff using Simple Good-Turing

I'm building an n-gram language model to predict the next word, I've implemented a simple Good-Turing smoothing on all my probabilities and have calculated the P0(mass probability of unseen event). I ...
5
votes
1answer
344 views

n-gram language model

At the end of the introduction of A Neural Probabilistic Language Model (Bengio et al. 2003), the following example is given: Having seen the sentence ...
1
vote
1answer
202 views

Curse of dimensionality with language models

In the seminal paper A Neural Probabilistic Language Model, Yoshua Bengio and his colleagues make the following point: If one wants to model the joint probability distribution of 10 consecutive ...
2
votes
1answer
103 views

How many words does the algorithm search through in Google Ngram? [closed]

When I run a query for "hers" in Google Ngram Viewer, I get back the word's frequency of occurrence as a percentage. We know the outcome percentage; what's the denominator on the other size? Is it 100 ...
10
votes
1answer
4k views

What are the pros and cons of applying pointwise mutual information on a word cooccurrence matrix before SVD?

One way to generate word embeddings is as follows: Get a corpora, e.g. "I enjoy flying. I like NLP. I like deep learning." Build the word cooccurrence matrix from it: Perform SVD on $X$, and keep ...
1
vote
1answer
605 views

Why two layers of matrix multiplication is needed in CBOW and Skip-gram model?

I found a nice tutorial here regarding CBOW and Skip-gram models for Word2Vec. I got the following doubt: Why two layers of matrix multiplication is need for CBOW and SKIP-GRAM ? As multiplying a ...