All Questions
Tagged with language-models recurrent-neural-network
13 questions
2
votes
1
answer
220
views
multiple likely ys for one instance of x: word prediction with LSTM
I have a ML project that is about predicting (suggesting) the next word based on the last n words, using LSTM. The output is a softmax dense layer the size of the vocabulary that shows the probability ...
- 51
3
votes
1
answer
4k
views
Use of ignore_index on CrossEntropyLoss() for text models
I have been using PyTorch's CrossEntropyLoss() on a Language Autoencoder. I noticed that most people use ignore_index for ignoring the pad token in loss calculation eg this.
From what I understand ...
- 33
0
votes
0
answers
22
views
Choosing a model for input: categorised, weighted sequence, output: binary variable
What would be an appropriate model for predicting a binary target variable, given a weighted sequence?
Sequences will be reasonably short, typically between ~ 1 and 5 elements. I have in the order of ...
- 101
2
votes
1
answer
831
views
BERT masking - why does it require sampling, and how does it mitigate the mismatch of the [MASK] token when fine-tuning?
I'm reading the BERT paper and jalammar's illustrative guide for BERT.
I don't understand 2 things about the method's crux - the masked language model:
why does masking requires us to sample (take ...
- 3,360
1
vote
1
answer
86
views
Do recurrent neural language models greedily model language probability?
Want to check my understanding of recurrent neural language models (in this case I'm working with a decoder in an encoder-decoder RNN but I don't think that matters significantly). I'm trying to ...
- 13
0
votes
1
answer
220
views
How to train a RNN language model?
I want to train a RNN-based language model from https://arxiv.org/pdf/1409.2329.pdf for next word prediction.
How to split the sentences from the dataset into input and ground truth during the ...
- 1
2
votes
1
answer
730
views
How to sample a language model?
I've successfully trained a language model using LSTMs. But I have a confusion about sampling.
On sampling, we generate a probability distribution at each time step. It will be of length vocabulary ...
- 123
1
vote
0
answers
35
views
Character level RNN for converting word forms
I want to build a char RNN to convert word form from one to another, for example, singular nouns such as lion to lions. However ...
3
votes
1
answer
220
views
How do we pass data to a RNN?
Let's say we have A1, A2, ... , Am different articles in the corpus and each of them has W1, W2, ....., Ww words. We are training a language model on them. Do we:
Scheme 1
Take the first batch of ...
- 2,693
2
votes
0
answers
451
views
Deciding on the training sequences for RNN/LSTM language model
In a character language model, text is seen as a stream of characters. Say we have a training text as a string s, with length ...
- 692
7
votes
2
answers
3k
views
Calculating test-time perplexity for seq2seq (RNN) language models
To compute the perplexity of a language model (LM) on a test sentence $s=w_1,\dots,w_n$ we need to compute all next-word predictions $P(w_1), P(w_2|w_1),\dots,P(w_n|w_1,\dots,w_{n-1})$.
My question ...
- 96
1
vote
1
answer
114
views
Under periodic BPTT, is softmax evaluated only at the end of the period?
Suppose I have a continuous sequence $X$ of words and I wish to train a RNN language model. According to [1], I would split $X$ into subsequences $X^{1..|X|/k_1}$ $k_1$ sized subsequences ($k_1$ is ...
- 210
5
votes
1
answer
8k
views
Advantage of character based language models over word based
Is there an intuition why character based models language bases models are preferred over word based. For example Karpathy builds his language model by predicting the next character in Karpathy Blog.
...
- 211