Questions tagged [language-models]

A statistical language model is a probability distribution over sequences of words.

Filter by
Sorted by
Tagged with
0
votes
0answers
5 views

Likelihood ratio test for language modeling

I am trying to use a likelihood ratio test to evaluate whether one language model is significantly better than another. (Note: In the example below, the language models should not be very different, ...
0
votes
0answers
12 views

Why can't standard conditional language models be trained left-to-right *and* right-to-left?

From the BERT paper: Unfortunately, standard conditional language models can only be trained left-to-right or right-to-left, since bidirectional conditioning would allow each word to indirectly “...
0
votes
0answers
17 views

How to compute the probability of a sentence with neural network language models

Many introductions to language models in NLP start by saying that the language modeling task aims to find the probability of a sequence of inputs (typically word tokens or characters). For instance ...
0
votes
0answers
12 views

How can an untrusted feature value affect the accuracy of a machine learning model?

I'm building a language detector model to classify between 2 languages given text image. The input will be the image, but I have another feature that can tell somehow the language (between 0,1) ...
1
vote
0answers
14 views

Contextual word embeddings to estimate likelihood of word given previous words in sentence?

I'd like to use contextual embeddings to estimate the likelihood of word n given the previous n-1 words in a sentence. Which pretrained models would allow me to do this (could I use something like ...
0
votes
0answers
7 views

What's the best way to store BERT training data (input IDs)

The tricky thing about the input IDs is what they're varying in length for each data sample, so regular hdf5 may not be ideal. Since Bert is so popular I am wondering if there's an established way to ...
0
votes
0answers
42 views

Perplexity for short sentences

I have a model that outputs short sentences and want to compare the quality of its outputs for different configurations by computing their perplexities using another model. I tried to use the gpt-2 ...
0
votes
0answers
16 views

Renormalising N-gram Probabilities to sum to 1

I am currently researching n-gram model smoothing using neural networks. In essence, my neural network produces a new probability for each n-gram in my model. I am using models in ARPA format. My ...
0
votes
1answer
35 views

Neural language model: Derivation of MLE

Recently, I studied NNLM and I saw the derivation of softmax using MLE: \begin{align} & \frac{\partial\log P(w_t\mid h)}{\partial\theta} \\[8pt] = {} & \frac{\partial \log \exp(s_\theta(w_t,...
0
votes
0answers
72 views

Denoising autoencoders vs masked approaches

I am not an expert in language modelling domain. There are mainly two approaches that are being used nowadays. Denoising autoencoders and ...
0
votes
0answers
98 views

How to Calculate Perplexity or cross entropy from Probability Distribution for a certain sentence?

I have a list of words starting with the letter "s" and their frequency count. From this, I'm trying to build a language model. I don't have the whole text, so I can't do the conditional probability, ...
0
votes
0answers
27 views

Good turing smoothing for unigram LM

I was wondering if it is at all possible to use good turing smoothing for unigram language model? I know that this smoothing technique helps distribute the weights from most occurring words to less ...
0
votes
1answer
50 views

How to train a RNN language model?

I want to train a RNN-based language model from https://arxiv.org/pdf/1409.2329.pdf for next word prediction. How to split the sentences from the dataset into input and ground truth during the ...
0
votes
1answer
37 views

Difference between ELMo and a normal deep network

I was reading up about ELMo and what I could gather was that we essentially combine the weights from different stacked lstm/gru layers for a given token as different layers are suited for different ...
0
votes
1answer
29 views

Likelihood for a test data (sequence of characters) given two unigram models

I would like to find the likelihood of a sequence of characters (the test data), given two unigram models. The sequence (test data) is: A B C B B The models ...
0
votes
1answer
20 views

Modeling words in a language based on their characters

I have different sets of strings, where I assume that each set follows some rules or patterns. For example, the first character must be a number, or the 3rd and the last characters must be the same, ...
0
votes
1answer
34 views

CMUDict normalized for word frequency

I am trying to train a neural grapheme to phoneme (G2P) model on CMUDict, but I find that pretty soon its loss is barely decreasing. Also, when I train the model on a different similar-sized dataset (...
1
vote
0answers
137 views

How to sample a language model?

I've successfully trained a language model using LSTMs. But I have a confusion about sampling. On sampling, we generate a probability distribution at each time step. It will be of length vocabulary ...
0
votes
1answer
299 views

Where can I find pre-trained language models in English and German?

Where can I find (more) pre-trained language models? I am especially interested in neural network based models for English and German. I am aware only of Language Model on One Billion Word Benchmark ...
0
votes
1answer
234 views

Perplexity of a Non-Statistical Language Model

I have a piece of software that, given a input phrase, returns an ordered list of the next most likely words (entire vocab is ordered 1 to n). This is essentially an Language Model with the exception ...
2
votes
1answer
1k views

How to handle big vocabulary size with keras tokenizer?

I am actually working on a neural language model developed with keras. I have an encoder and a decoder and the output of the decoder is a dense vector on the vocabulary..so quite big depending on the ...
1
vote
1answer
78 views

How does clustering improve a language model?

This article describes a hierarchical clustering algorithm which clusters the words within a vocabulary based on their similarity, in order to improve a language model (in the article, n-grams). How ...
4
votes
0answers
36 views

natural language understanding algorithms [closed]

I am doing some research into how smart personal assistants work like siri, alexa etc. I have found that it is using automatic speech recognition to turn the speech into weighted text form and then ...
0
votes
0answers
92 views

Normalize probability distribution by variance of each class

I have several topics that I will collect language data for. Using Mturk I will ask responders to write sentences for each topic. The sentences will be used to train a language model. Language models ...
1
vote
2answers
64 views

What are commonly used methods to represent a document by a vector?

Methods that I know of Bag of words + weighting: tf-idf, bm25 Topic models: LSA, LDA Word/sentence/document embedding Are there other commonly used methods to represent a document by a vector?
1
vote
1answer
110 views

How to compute context-independent token representations in a biLM?

I've been reading this paper on ELMo word representations. For context, here's my understanding of the standard bi-directional language model (biLM) thus far: Given a sequence of tokens $(t_{1}, ...
1
vote
1answer
24 views

What the 'N' means on the following N-gram approximation

What the 'N'(capital) means in the following N-gram approximation to the conditional probability of the next word in a sequence ?
1
vote
0answers
27 views

Character level RNN for converting word forms

I want to build a char RNN to convert word form from one to another, for example, singular nouns such as lion to lions. However ...
0
votes
1answer
595 views

Perplexity calculation with neural nets

I am having troubles understanding which formula to use to calculate perplexity of a neural language model. Various places online on the forums people suggest using 2^(cross-entropy) measure, which is ...
2
votes
1answer
827 views

Skip-gram algorithm confusion

As a newbie to NLP, I am (deeply) confused by the middle step in the following diagram explaining the skip-gram algorithm. The video where this diagram was presented can be found at: https://www....
2
votes
1answer
129 views

How do we pass data to a RNN?

Let's say we have A1, A2, ... , Am different articles in the corpus and each of them has W1, W2, ....., Ww words. We are training a language model on them. Do we: Scheme 1 Take the first batch of ...
1
vote
1answer
69 views

Finding a mixture of 1st and 0'th order Markov models that is closest to an empirical distribution

I am interested in finding the distribution "$p^*$" closest to an empirical distribution $\hat{p}$ where $p^*$ is a mixture of first and zeroth order Markov models. That is, I want to find $$ p^* = \...
0
votes
1answer
425 views

Bayesian smoothing using Dirichlet prior : why not MAP?

I am reading about smoothing methods for language model ( I am working on unigram model). If you are not familiar with unigram model, it is closely related to multinomial distribution (with the ...
0
votes
0answers
51 views

Decide threshold for LM to filter out ill-formed sentence

I have a corpus of sentences. This corpus contains well-formed as well as ill-formed sentences. I want to filter out ill-formed sentences from the corpus. Ill-formed sentences are of two types : ...
4
votes
1answer
1k views

Language Model compare probability scores between Length varying sentence

My question is : How can I compare Language Model(LM) score for two sentences with different lengths ? Probabilities are < 1...
1
vote
1answer
841 views

what are hidden states in HMM based language model?

There are several ways to build language models, n-gram based models are straightforward, but for the language models built on HMMs, what are hidden states and what are observations?
1
vote
2answers
181 views

Language model created with SRILM does not sum to 1

I created an n-gram language model on the Penn Treebank using the following command: ...
0
votes
0answers
49 views

Classifying Sentences when outcome is a sentence

I am completely new to text classification and facing the following problem, I want to classify sentences and have a labeled dataset, the thing is that my Y variable is itself a sentence, hence I ...
3
votes
2answers
385 views

Language Modelling using Neural Networks

I plan to make a Language Model in Python using Neural Networks. I've read that Neural Networks need vectors as input. One common vector representation in NLP is the Bag of Words model. Given a corpus ...
1
vote
0answers
250 views

Deciding on the training sequences for RNN/LSTM language model

In a character language model, text is seen as a stream of characters. Say we have a training text as a string s, with length ...
2
votes
0answers
852 views

Computing Training Set Perplexity of a Neural Language Model: Too low values

I am implementing a Language Model based on a Deep Learning architecture (RNN+Softmax). The cost function I am using is the cross-entropy between the vector of probabilities at the softmax layer and ...
2
votes
2answers
994 views

Language Detection with CLD2 with Mixed Inputs in long documents

Internals Recap. CLD2 is a Naïve Bayesian classifier, trained on documents of mean size of 200 characters, trained on a corpus of 100M scraped and human expert selected web pages. When working on ...
6
votes
2answers
1k views

Calculating test-time perplexity for seq2seq (RNN) language models

To compute the perplexity of a language model (LM) on a test sentence $s=w_1,\dots,w_n$ we need to compute all next-word predictions $P(w_1), P(w_2|w_1),\dots,P(w_n|w_1,\dots,w_{n-1})$. My question ...
1
vote
1answer
1k views

A simple numerical example for Kneser-Ney Smoothing

I'm working in a project trying to implement the Kneser-Key algorithm. I think I got up to the step of implementing this formula for bigrams: $P_{(KN)}(w_i|w_{i-1}) = \frac{max(c(w_{-1}, w_{1}) - \...
1
vote
1answer
4k views

How to calculate the perplexity of test data versus language models

I have been working on an assignment where I train upon 3 corpora in 3 separate languages, and then I read in a set of sentences and use a number of models to determine the most likely language for ...
1
vote
0answers
396 views

Feature extraction from strings

We are given a big collection of strings, and an intensity associated with each string in the collection. In a sense, a 'distribution' on the dictionary. We are given that the intensity of each string ...
3
votes
2answers
3k views

Understanding Add-1/Laplace smoothing with bigrams

I am working through an example of Add-1 smoothing in the context of NLP Say that there is the following corpus (start and end tokens included) ...
1
vote
1answer
94 views

Under periodic BPTT, is softmax evaluated only at the end of the period?

Suppose I have a continuous sequence $X$ of words and I wish to train a RNN language model. According to [1], I would split $X$ into subsequences $X^{1..|X|/k_1}$ $k_1$ sized subsequences ($k_1$ is ...
6
votes
2answers
1k views

Neural network language model - prediction for the word at the center or the right of context words

Neural network language model - prediction for the word at the center or the right of context words? On Bengio's paper, the model predicts probability by n words for the next word, like predicting ...
2
votes
1answer
1k views

In Kneser Ney smoothing, how to implement the recursion in the formula?

I'm working in a project trying to implement the Kneser-Key algorithm. I think I got up to the step of implementing this formula for bigrams: $P_{(KN)}(w_i|w_{i-1}) = \frac{max(c(w_{-1}, w_{1}) - \...