All Questions
Tagged with language-models word-embeddings
12 questions
0
votes
0
answers
248
views
Is it necessary to use attention mask for mean pooling for BERT?
I am working on a project involving the analysis of clinical texts using the "emilyalsentzer/Bio_ClinicalBERT" model from Hugging Face's transformers library. My goal is to extract ...
- 138
0
votes
1
answer
34
views
Self-supervised Target Definition in the Original Neural Language Model by Bengio et al (2003)
I understand how later neural language models (such as those used in the Word2Vec papers) framed the language modelling problem in a self-supervised way by learning to predict the next word (or any ...
- 1,046
1
vote
1
answer
340
views
Feedforward Neural Net Language Model - computational complexity (word2vec)
While reading this paper on word2vec, I came around the following description of a feedforward Neural Net Language model (NNLM):
It consists of input, projection, hidden and output layers. At the ...
- 41
0
votes
0
answers
22
views
Choosing a model for input: categorised, weighted sequence, output: binary variable
What would be an appropriate model for predicting a binary target variable, given a weighted sequence?
Sequences will be reasonably short, typically between ~ 1 and 5 elements. I have in the order of ...
- 101
2
votes
1
answer
515
views
What's the best practice for dealing with OOV characters?
I have read on the advantages of using character-level language models over word-level ones. In particular, you don't have to deal with the problem of out of vocabulary (OOV) words, since characters ...
- 33
3
votes
0
answers
1k
views
Different ways to calculate pointwise mutual information for word co-occurrence [closed]
I have a (very) small corpus of documents. As a representative example: 450 documents, 280000 total word count.
I am calculating Positive Pointwise Mutual Information (PPMI) between a selection of ...
- 133
1
vote
0
answers
31
views
Contextual word embeddings to estimate likelihood of word given previous words in sentence?
I'd like to use contextual embeddings to estimate the likelihood of word n given the previous n-1 words in a sentence. Which pretrained models would allow me to do this (could I use something like ...
- 11
0
votes
1
answer
27
views
Modeling words in a language based on their characters
I have different sets of strings, where I assume that each set follows some rules or patterns. For example, the first character must be a number, or the 3rd and the last characters must be the same, ...
- 13
2
votes
1
answer
189
views
How to compute context-independent token representations in a biLM?
I've been reading this paper on ELMo word representations.
For context, here's my understanding of the standard bi-directional language model (biLM) thus far:
Given a sequence of tokens $(t_{1}, ...
1
vote
1
answer
810
views
Does the skipgram language model try to predict all context words at the same time?
In the skipgram language model (Mikolov et al., 2013), a neural network with one hidden layer tries to predict surrounding words from current words of the corpus. After training, the hidden activation ...
- 990
16
votes
1
answer
10k
views
What are the pros and cons of applying pointwise mutual information on a word cooccurrence matrix before SVD?
One way to generate word embeddings is as follows (mirror):
Get a corpora, e.g. "I enjoy flying. I like NLP. I like deep learning."
Build the word cooccurrence matrix from it:
Perform SVD on $X$, ...
- 47.6k
11
votes
2
answers
3k
views
Question about Continuous Bag of Words
I'm having trouble understanding this sentence:
The first proposed architecture is similar to the feedforward NNLM,
where the non-linear hidden layer is removed and the projection
layer is ...
- 323