Questions tagged [natural-language]
Natural Language Processing is a set of techniques from linguistics, artificial intelligence, machine learning and statistics that aim at processing and understanding human languages.
1,156
questions
0
votes
0
answers
17
views
Regression with text data
My goal is to create a regression model with text data where encoded text predicts a value, (news headlines, or article summaries, predicting number of clicks). The y is very left-skewed (few articles ...
- 31
0
votes
0
answers
11
views
Classification in BERT - why not use class as a feature?
I am currently following this post, which details how BERT was trained. I had a few questions about the classification task:
In the post, it mentions that the authors of BERT decided to add ...
- 269
0
votes
0
answers
8
views
How does Kneser Ney estimate ngrams with BOS without dividing by zero?
The recursive formula for (unmodified) Kneser Ney smoothing is (per Jurafsky08 3.40)
$P_{KN}(w_i | h) = \frac{\text{max}(c_{KN}(h\ w_i) - d, 0)}{\sum_v c_{KN}(h \ v )} + \lambda(h) P_{KN}(w_i | h)$
...
- 1,366
0
votes
0
answers
10
views
Training an Image Captioning Model with variable number of captions per image
I am following this guide for training an Image Captioning model
It uses a dataset which always has 5 captions per image. My dataset greatly varies how many captions I have per image from 1-42. This ...
- 133
2
votes
0
answers
49
views
Can I calculate the significance of the number of deponent verbs with a certain feature like this?
In a language like Ancient Greek, verbal forms are marked for voice (active/middle/passive). Deponent verbs are verbs that exist only in the middle (or passive) voice, but appear to have an active ...
- 121
1
vote
1
answer
20
views
Gradient Clipping of Vanilla RNNs vs LSTMs
I am doing an online course that states that the reason we use LSTMs and similar variations of vanilla RNNs is because of the vanishing/exploding gradients problems with vanilla RNNs.
However, an ...
- 11
0
votes
1
answer
20
views
Why is the WordPiece algorithm implemented according to the maximum mutual information?
WordPiece is a subword segmentation algorithm in the field of natural language processing. Different from BPE, WordPiece will select a pair with the largest mutual information to merge each time, and ...
1
vote
0
answers
24
views
Does skipgram model uses backpropagation?
I just started to get interested in natural language processing and I was trying to understand the skipgram model from word2vec. I was reading this interesting website. However, in the mentioned ...
- 111
3
votes
3
answers
476
views
Countering t-test "any feature is significant" results for large sample size datasets
I'm doing some analysis over natural language data, which basically entails:
Computing some feature over all samples.
Evaluating if this feature statistically significantly discriminates between ...
- 31
0
votes
0
answers
20
views
How is metadata represented in sentiment analysis?
There are papers on semantic analysis using metadata such as "Sentiment Classification on Steam Reviews" (https://cs229.stanford.edu/proj2017/final-reports/5244171.pdf) and "Detecting ...
- 1
0
votes
0
answers
15
views
Implementation of spBLEU
I was looking for a way to compute statistics for evaluation metrics for language translation models and I came across spBLEU. I can’t find any implementations/examples that would help me start. Does ...
- 1
0
votes
0
answers
15
views
How can BERT/Transformer models accept input batches of different sizes?
I understand that all inputs in a batch need to be of the same size. However, it seems BERT/Transformers models can accept batches with different sizes as input.
How is that possible? I thought we ...
- 657
0
votes
0
answers
9
views
How word2vec de-embeds the special names in language models which output text
I am new to nlp field. I have some questions about word2vec embeddings. as I know they have a fixed size dictionary of vocabs. so definitely there some words which is not in that predefined dictionary ...
- 169
0
votes
1
answer
64
views
What is the Llama2 number of steps? [closed]
Llama2 is pretrained with 2 trillion of tokens: $2\times10^9$, and its batch size is of $4\times 10^6$.
We can calculate the number of steps (times we upgrade the parameters) per epoch as follows:
$$\...
- 3
1
vote
0
answers
15
views
Does it make sense to perform Domain Adaptation before Transfer Learning?
Suppose I would like to do extractive question answering on scientific literature. I'm interested in using BERT which was pretrained on Wiki and Bookcorpus. I see two routes here:
1. Fine-tune BERT on ...
- 11
0
votes
0
answers
14
views
Would it be wise to feed k-means results into a cmeans?
I am conducting a cluster analysis of documents using document embeddings as the input to the algorithms. One of the problems I am coming across is that in reality there are documents that belong to ...
- 31
0
votes
0
answers
20
views
What are more sophisticated measures for word similarity than cosine similarity?
I am trying to compute measures of word/sentence similarity from word embeddings that I would like to use for classification. However, cosine similarity often runs counter to how I as a human would ...
- 101
0
votes
0
answers
10
views
How does Doc2Vec handle documents where the length is less than the window size?
I am using doc2vec to vectorise documents that average in length 10 words (PV-DBOW implementation of the algorithm). I am wondering how doc2vec handles cases where the number of words in a document is ...
- 31
0
votes
0
answers
10
views
How do Doc2Vec PV-DBOW iterations/epoch's work?
Please correct me if I'm wrong but as I understand it, one iteration (?) of doc2vec takes each document and predicts a series of randomly sampled context words individually, feeding the errors from ...
- 31
0
votes
1
answer
17
views
Considering weights right of the embeddings layer aren't used in Doc2Vec/Word2Vec, is the informative capacity of the embeddings not strongly reduced?
In an extreme (and probably impossible) example, could you not end up with all the power for the prediction being contained in the weights to the right of the embeddings layer?...and thus the matrix ...
- 31
1
vote
1
answer
54
views
How are vector values assigned initially in Word2Vec and how are they changed with iterations of the algorithm?
I am new to NLP and I'm not fully grasping how word2vec works. I understand that it aims to predict a word given its context or a context given a word but I'm not sure how the initial vector values ...
- 31
0
votes
1
answer
49
views
Using Word Embeddings in Clustering and Topic Modelling
I am new to the field of NLP and would appreciate any guidance please. I am trying to understand how word embeddings can be used in clustering and topic modelling. If I create word embeddings for ...
- 31
0
votes
0
answers
9
views
ValueError while fitting a neural networks model on Consumer Complaints data
I am trying to build a keras tensorflow neural network model. I have never built a NN model prior so facing challenges with some very basic error.
I Was able to build the model using following code. I ...
0
votes
1
answer
50
views
How does training word embeddings bring similar words closer together?
How does training of word embeddings lead to the clustering of similar words in the embedding space? What causes that effect?
- 433
1
vote
0
answers
122
views
How to determine EC2 instance type and memory for LLM inference endpoint [closed]
I am trying to estimate the costs required for hosting a fine tuned large language model for real time inference. There will be 100s of users querying the endpoint concurrently for multiple use cases ...
- 111
1
vote
0
answers
30
views
Machine learning and Natural Language Processing Algorithms for Indian Surnames Homophones [closed]
Homophones
Indian Surnames List
English last names
Can machine learning, Natural Language Processing (NLP), Artificial intelligence assist in classifying , interpreting and specifying the differences ...
- 119
0
votes
1
answer
29
views
Creating a morphology tagging/labeling model
I had an idea of building a model using machine learning or deep learning in order to perform morphological tagging/labeling on untagged/unlabeled data.
I have a lot of tagged/labeled data (about 30,...
0
votes
0
answers
38
views
Training a variable-length document-level LSTM on variable-length sentence-level data using ngrams and majority voting
Let's assume that I want to train a classification model for document-level input. The input is sequential (a sequence of tokens within a text). Documents may vary in length (i.e., one or multiple ...
0
votes
1
answer
38
views
Is concatenating a single integer sufficient for encoding positional embeddings in transformer models?
In transformer models, positional embeddings are commonly used to encode the positional information of words in a sequence. While sinusoidal positional embeddings are often employed, I'm curious about ...
- 433
1
vote
1
answer
43
views
Input non-sequential data of arbitrary size to network
I have a case where I want to feed a network with polylines of data. The problem is that the input can be any number of polylines and the polylines can consist of any number of points. If we instead ...
- 61
2
votes
0
answers
84
views
Attention is All You Need: How to calculate params number of the models?
I want to re-calculate the last column of Table 3 of Attention is All You Need,
i.e. number of params in the models.
But numbers from my calculation do not match.
Model
Params from Table 3 ($\times ...
- 21
1
vote
0
answers
64
views
Sentiment Analysis with a Continuos Output Labels
Problem Setting/Context:
I have feedback(each feedback has multiple sentences) associated with different products(you can safely assume that a feedback talks about one single product), I need to ...
- 21
1
vote
0
answers
10
views
Accounting for edge cases without training on the test set
I'm fine-tuning a large language model to predict binary sentiment, where a false negative is far more costly for my use case than a false positive. I've used weighted cross-entropy to account for ...
0
votes
1
answer
107
views
Why the Transformer model does not require negative sampling but word2vec does?
Both word2vec and transformer model compute a SOFTMAX function over the words/tokens on the output side.
For word2vec models, negative sampling is used for computational reasons:
Is negative sampling ...
- 2,335
1
vote
0
answers
60
views
Why does the best performing adapter-based parameter-efficient fine-tuning depend on the language model being fine-tuned?
https://arxiv.org/abs/2304.01933 shows that the best performing adapter-based parameter-efficient fine-tuning depends on the language model being fine-tuned:
E.g., LORA is the best adapter for LlaMa-...
- 45.5k
0
votes
0
answers
8
views
How can I produce word embeddings from code-mixed data using Facebook MUSE?
I am doing sarcasm detection using code mixed data. Sentences are in English and Hindi written in English alphabets. Is there a way to apply Facebook MUSE to generate word embeddings out of this data?
- 119
0
votes
0
answers
466
views
How to calculate loss in pre-training gpt-2
As I know BERT model that calculates loss by compare between an original word that was masked with the predicted result for retrieve the loss and update the weight in pre-train model.
But GPT-2 uses ...
- 1
0
votes
0
answers
86
views
Perplexity with log probabilities
I want to calculate the perplexity of my language model. To avoid underflow, I stored the log of the probabilities. Since the probabilities are between 0 and 1, the log of them is negative. So when I ...
- 111
0
votes
0
answers
22
views
Training a text model for similarity
I'm training a text similarity model using a two tower approach. The data I'm dealing with has a lot of unusual words that are important (names of people, places) that also don't appear in any pre-...
- 111
0
votes
0
answers
16
views
Why are my deep learning models giving unreasonably high accuracy on test data?
I'm trying to do sarcasm detection on Twitter data to replicate the results mentioned in this paper. Binary classification problem. For that I used a separate set of unlabeled tweets to create the ...
- 119
1
vote
1
answer
16
views
Best Way to do Hyperparam Search and Cross-Validation
I'm making experiments to evaluate language models to brazilian portuguese datasets.
So, i've made so each dataset is divided in 10 parts, I want to use cross-validation to determine the model's ...
0
votes
0
answers
40
views
Can the same training data be used for MLM and fine-tuning of a transformer model?
Can I use the same training and validation data to perform MLM and train the weights of a classification head?
Here is the background of my specific problem:
The problem is a binary classification ...
- 31
0
votes
1
answer
59
views
Estimating exponent of Zipf distribution using MLE vs fitting linear regression on log-transformed rank and frequency data
I'm having trouble understanding why I get radically different results if I try to find the parameter of a Zipf distribution when I use the methods proposed by Clauset et al. (2009) as opposed to ...
- 65
0
votes
1
answer
27
views
Smallest set of words contained in a number of documents
I have a collection of text documents. I would like to find the smallest set of words such that searching by those words allows discovering each document.
It is quite natural to describe this data as ...
- 969
1
vote
1
answer
226
views
How do I go from embeddings to queries, keys and values in the Transformer model?
I am trying to implement Attention Is All You Need paper from scratch in PyTorch. So far, I implemented the Scaled Dot-Product Attention layer and the Multi-Head Attention layer. As I began to write ...
1
vote
0
answers
11
views
How can I use an ontology schema to classify triples? [closed]
I have extracted some triples from text using REBEL an information extraction technic. How do I automatically establish where these triples fit into my pre-existing ontology? How can I use the ...
- 11
3
votes
1
answer
584
views
How to compare the semantic similarity of text generated by large language models (GPT-3, BLOOM etc) to reference text?
Large language models, such as GPT-3, BLOOM etc, can generate open-ended text. Say I want to prompt these models to answer a question. How can I compare the semantic similarity of the answer it ...
- 31
2
votes
0
answers
54
views
Why is less than 0.1 considered too low learning rate in this example
I watched this (with time marker to important moment https://youtu.be/TCH_1BHY58I?t=3040) video of Andrej Karpathy explaining some language model, and in ~50:40, he explains how to pick good initial ...
- 21
1
vote
1
answer
60
views
Intuitive difference between NN and attention for text prediction
If your task is to predict $t_{n+1}$ given tokens $(t_1,...,t_n)$, you could do two things:
Straight NN - feed $t=(t_1,...,t_n)$ into a neural network as an n-dimensional input and train it on ...
1
vote
0
answers
26
views
Attention mechanism with different hidden states length?
i have two layer with different layer sizes (hidden states) how can i perform encoder decoder type of attention on these layers if the layer sizes are different? because i will do dot product, how?
...
- 11