Questions tagged [language-models]
A statistical language model is a probability distribution over sequences of words.
110 questions
1
vote
1
answer
86
views
Do recurrent neural language models greedily model language probability?
Want to check my understanding of recurrent neural language models (in this case I'm working with a decoder in an encoder-decoder RNN but I don't think that matters significantly). I'm trying to ...
- 13
8
votes
1
answer
2k
views
Why can't standard conditional language models be trained left-to-right *and* right-to-left?
From the BERT paper:
Unfortunately, standard conditional language models can only be trained left-to-right or right-to-left, since bidirectional conditioning would allow each word to indirectly “...
- 1,386
0
votes
0
answers
13
views
How can an untrusted feature value affect the accuracy of a machine learning model?
I'm building a language detector model to classify between 2 languages given text image. The input will be the image, but I have another feature that can tell somehow the language (between 0,1) ...
- 101
1
vote
0
answers
31
views
Contextual word embeddings to estimate likelihood of word given previous words in sentence?
I'd like to use contextual embeddings to estimate the likelihood of word n given the previous n-1 words in a sentence. Which pretrained models would allow me to do this (could I use something like ...
- 11
1
vote
0
answers
516
views
Perplexity for short sentences
I have a model that outputs short sentences and want to compare the quality of its outputs for different configurations by computing their perplexities using another model.
I tried to use the gpt-2 ...
- 11
0
votes
1
answer
106
views
Neural language model: Derivation of MLE
Recently, I studied NNLM and I saw the derivation of softmax using MLE:
\begin{align}
& \frac{\partial\log P(w_t\mid h)}{\partial\theta} \\[8pt]
= {} & \frac{\partial \log
\exp(s_\theta(w_t,...
- 171
5
votes
1
answer
1k
views
Denoising autoencoders vs masked approaches
I am not an expert in language modelling domain.
There are mainly two approaches that are being used nowadays. Denoising autoencoders and ...
- 378
0
votes
1
answer
220
views
How to train a RNN language model?
I want to train a RNN-based language model from https://arxiv.org/pdf/1409.2329.pdf for next word prediction.
How to split the sentences from the dataset into input and ground truth during the ...
- 1
0
votes
1
answer
62
views
Difference between ELMo and a normal deep network
I was reading up about ELMo and what I could gather was that we essentially combine the weights from different stacked lstm/gru layers for a given token as different layers are suited for different ...
- 101
0
votes
1
answer
195
views
Likelihood for a test data (sequence of characters) given two unigram models
I would like to find the likelihood of a sequence of characters (the test data), given two unigram models.
The sequence (test data) is:
A B C B B
The models ...
- 101
0
votes
1
answer
27
views
Modeling words in a language based on their characters
I have different sets of strings, where I assume that each set follows some rules or patterns. For example, the first character must be a number, or the 3rd and the last characters must be the same, ...
- 13
0
votes
1
answer
112
views
CMUDict normalized for word frequency
I am trying to train a neural grapheme to phoneme (G2P) model on CMUDict, but I find that pretty soon its loss is barely decreasing. Also, when I train the model on a different similar-sized dataset (...
- 101
2
votes
1
answer
730
views
How to sample a language model?
I've successfully trained a language model using LSTMs. But I have a confusion about sampling.
On sampling, we generate a probability distribution at each time step. It will be of length vocabulary ...
- 123
1
vote
1
answer
384
views
Where can I find pre-trained language models in English and German?
Where can I find (more) pre-trained language models? I am especially interested in neural network based models for English and German.
I am aware only of Language Model on One Billion Word Benchmark ...
0
votes
1
answer
519
views
Perplexity of a Non-Statistical Language Model
I have a piece of software that, given a input phrase, returns an ordered list of the next most likely words (entire vocab is ordered 1 to n). This is essentially an Language Model with the exception ...
- 101
3
votes
1
answer
4k
views
How to handle big vocabulary size with keras tokenizer?
I am actually working on a neural language model developed with keras.
I have an encoder and a decoder and the output of the decoder is a dense vector on the vocabulary..so quite big depending on the ...
- 31
1
vote
1
answer
206
views
How does clustering improve a language model?
This article describes a hierarchical clustering algorithm which clusters the words within a vocabulary based on their similarity, in order to improve a language model (in the article, n-grams).
How ...
- 359
3
votes
0
answers
39
views
natural language understanding algorithms [closed]
I am doing some research into how smart personal assistants work like siri, alexa etc. I have found that it is using automatic speech recognition to turn the speech into weighted text form and then ...
- 31
0
votes
0
answers
98
views
Normalize probability distribution by variance of each class
I have several topics that I will collect language data for. Using Mturk I will ask responders to write sentences for each topic. The sentences will be used to train a language model. Language models ...
- 51
2
votes
2
answers
120
views
What are commonly used methods to represent a document by a vector?
Methods that I know of
Bag of words + weighting: tf-idf, bm25
Topic models: LSA, LDA
Word/sentence/document embedding
Are there other commonly used methods to represent a document by a vector?
- 16.8k
2
votes
1
answer
189
views
How to compute context-independent token representations in a biLM?
I've been reading this paper on ELMo word representations.
For context, here's my understanding of the standard bi-directional language model (biLM) thus far:
Given a sequence of tokens $(t_{1}, ...
1
vote
1
answer
42
views
What the 'N' means on the following N-gram approximation
What the 'N'(capital) means in the following N-gram
approximation to the conditional probability of the next word in a sequence ?
- 111
1
vote
0
answers
35
views
Character level RNN for converting word forms
I want to build a char RNN to convert word form from one to another, for example, singular nouns such as lion to lions. However ...
0
votes
1
answer
1k
views
Perplexity calculation with neural nets
I am having troubles understanding which formula to use to calculate perplexity of a neural language model. Various places online on the forums people suggest using 2^(cross-entropy) measure, which is ...
- 101
5
votes
1
answer
1k
views
Skip-gram algorithm confusion
As a newbie to NLP, I am (deeply) confused by the middle step in the following diagram explaining the skip-gram algorithm. The video where this diagram was presented can be found at:
https://www....
- 377
3
votes
1
answer
220
views
How do we pass data to a RNN?
Let's say we have A1, A2, ... , Am different articles in the corpus and each of them has W1, W2, ....., Ww words. We are training a language model on them. Do we:
Scheme 1
Take the first batch of ...
- 2,683
2
votes
1
answer
88
views
Finding a mixture of 1st and 0'th order Markov models that is closest to an empirical distribution
I am interested in finding the distribution "$p^*$" closest to an empirical distribution $\hat{p}$ where $p^*$ is a mixture of first and zeroth order Markov models. That is, I want to find $$
p^* = \...
- 751
2
votes
1
answer
1k
views
Bayesian smoothing using Dirichlet prior : why not MAP?
I am reading about smoothing methods for language model ( I am working on unigram model).
If you are not familiar with unigram model, it is closely related to multinomial distribution (with the ...
- 57
0
votes
0
answers
75
views
Decide threshold for LM to filter out ill-formed sentence
I have a corpus of sentences. This corpus contains well-formed as well as ill-formed sentences. I want to filter out ill-formed sentences from the corpus.
Ill-formed sentences are of two types :
...
4
votes
1
answer
3k
views
Language Model compare probability scores between Length varying sentence
My question is : How can I compare Language Model(LM) score for two sentences with different lengths ?
Probabilities are < 1...
2
votes
1
answer
2k
views
what are hidden states in HMM based language model?
There are several ways to build language models, n-gram based models are straightforward, but for the language models built on HMMs, what are hidden states and what are observations?
- 33
2
votes
2
answers
442
views
Language model created with SRILM does not sum to 1
I created an n-gram language model on the Penn Treebank using the following command:
...
- 21
0
votes
0
answers
51
views
Classifying Sentences when outcome is a sentence
I am completely new to text classification and facing the following problem, I want to classify sentences and have a labeled dataset, the thing is that my Y variable is itself a sentence, hence I ...
- 103
3
votes
2
answers
453
views
Language Modelling using Neural Networks
I plan to make a Language Model in Python using Neural Networks. I've read that Neural Networks need vectors as input. One common vector representation in NLP is the Bag of Words model. Given a corpus ...
- 53
2
votes
0
answers
451
views
Deciding on the training sequences for RNN/LSTM language model
In a character language model, text is seen as a stream of characters. Say we have a training text as a string s, with length ...
- 692
2
votes
0
answers
2k
views
Computing Training Set Perplexity of a Neural Language Model: Too low values
I am implementing a Language Model based on a Deep Learning architecture (RNN+Softmax).
The cost function I am using is the cross-entropy between the vector of probabilities at the softmax layer and ...
- 71
3
votes
2
answers
2k
views
Language Detection with CLD2 with Mixed Inputs in long documents
Internals Recap. CLD2 is a Naïve Bayesian classifier, trained on documents of mean size of 200 characters, trained on a corpus of 100M scraped and human expert selected web pages.
When working on ...
- 153
7
votes
2
answers
3k
views
Calculating test-time perplexity for seq2seq (RNN) language models
To compute the perplexity of a language model (LM) on a test sentence $s=w_1,\dots,w_n$ we need to compute all next-word predictions $P(w_1), P(w_2|w_1),\dots,P(w_n|w_1,\dots,w_{n-1})$.
My question ...
- 96
1
vote
1
answer
4k
views
A simple numerical example for Kneser-Ney Smoothing
I'm working in a project trying to implement the Kneser-Key algorithm. I think I got up to the step of implementing this formula for bigrams:
$P_{(KN)}(w_i|w_{i-1}) = \frac{max(c(w_{-1}, w_{1}) -
\...
2
votes
1
answer
6k
views
How to calculate the perplexity of test data versus language models
I have been working on an assignment where I train upon 3 corpora in 3 separate languages, and then I read in a set of sentences and use a number of models to determine the most likely language for ...
- 173
1
vote
0
answers
411
views
Feature extraction from strings
We are given a big collection of strings, and an intensity associated with each string in the collection. In a sense, a 'distribution' on the dictionary. We are given that the intensity of each string ...
5
votes
2
answers
12k
views
Understanding Add-1/Laplace smoothing with bigrams
I am working through an example of Add-1 smoothing in the context of NLP
Say that there is the following corpus (start and end tokens included)
...
- 173
1
vote
1
answer
114
views
Under periodic BPTT, is softmax evaluated only at the end of the period?
Suppose I have a continuous sequence $X$ of words and I wish to train a RNN language model. According to [1], I would split $X$ into subsequences $X^{1..|X|/k_1}$ $k_1$ sized subsequences ($k_1$ is ...
- 210
7
votes
2
answers
1k
views
Neural network language model - prediction for the word at the center or the right of context words
Neural network language model - prediction for the word at the center or the right of context words?
On Bengio's paper, the model predicts probability by n words for the next word, like predicting ...
- 848
2
votes
1
answer
2k
views
In Kneser Ney smoothing, how to implement the recursion in the formula?
I'm working in a project trying to implement the Kneser-Key algorithm. I think I got up to the step of implementing this formula for bigrams:
$P_{(KN)}(w_i|w_{i-1}) = \frac{max(c(w_{-1}, w_{1}) -
\...
- 123
5
votes
1
answer
8k
views
Advantage of character based language models over word based
Is there an intuition why character based models language bases models are preferred over word based. For example Karpathy builds his language model by predicting the next character in Karpathy Blog.
...
- 211
1
vote
1
answer
810
views
Does the skipgram language model try to predict all context words at the same time?
In the skipgram language model (Mikolov et al., 2013), a neural network with one hidden layer tries to predict surrounding words from current words of the corpus. After training, the hidden activation ...
- 990
1
vote
2
answers
2k
views
Kneser-Ney for unigrams?
I was wondering if it is at all possible to use Kneser-Ney to smooth word unigram probabilites?
The basic idea behind back-off is to use (n-1)-gram frequencies when an n-gram has 0 count. This is ...
- 202
2
votes
0
answers
272
views
Alpha on Katz Backoff using Simple Good-Turing
I'm building an n-gram language model to predict the next word, I've implemented a simple Good-Turing smoothing on all my probabilities and have calculated the P0(mass probability of unseen event).
I ...
- 41
6
votes
1
answer
525
views
n-gram language model
At the end of the introduction of A Neural Probabilistic Language Model (Bengio et al. 2003), the following example is given:
Having seen the sentence ...
- 6,217