Skip to main content

Questions tagged [natural-language]

Natural Language Processing is a set of techniques from linguistics, artificial intelligence, machine learning and statistics that aim at processing and understanding human languages.

Filter by
Sorted by
Tagged with
1 vote
0 answers
6 views

NER With Custom Tags, How to Approach

I am building a "field tagger" for documents. Basically, a document, in my case something like a proposal or sales quote, would have a bunch of entities scattered throughout it, and we want ...
redbull_nowings's user avatar
0 votes
0 answers
26 views

Normalizing the embedding space of an encoder language model with respect to categorical data

Suppose we have a tree/hierarchy of categories (e.g. categories of products in an e-commerce website), each node being assigned a title. Assume that the title of each node is semantically accurate, ...
mtcicero's user avatar
  • 123
0 votes
0 answers
9 views

Why learn an embedding before self attention when training transformers?

I understand that self-attention layers learn the "role" of a word in a sentence while embedding layers learn the relationship between the words. But I am not totally convinced that a self-...
Nicolas Johnson's user avatar
0 votes
0 answers
12 views

Log-likelihood calculation for unigrams

I am calculating the log-likelihood for each unigram that I generated by using the CountVectorizer to see each unigram's importance. However, I got all the positive value after calculating the log-...
Nick's user avatar
  • 1
4 votes
2 answers
534 views

Overfitting in randomForest model in R, WHY?

I am trying to train a Random Forest model in R for sentiment analysis. The model works with tf-idf matrix and learns from it how to classify a review, in positive or negative. Positive ones are ...
Anisa's user avatar
  • 43
0 votes
0 answers
20 views

Where does the equation $ C = 6 \times N \times T $ come from for Large Language Models, especially with a simple explanation for both passes?

Why $ C = 6 \times N \times T $? I'm trying to understand the computational steps specifically during the backward pass of neural networks in relation to the widely cited formula ( C = 6 \times N \...
Charlie Parker's user avatar
0 votes
0 answers
18 views

Can 3D convolutions appropriately capture a frozen embedding space?

My project is a strange combination of NLP and Computer Vision. I have datapoints of 3D tensor where each element is a token in an NLP vocabulary. The vocabulary is around 1000 unique "words"...
schmixi's user avatar
  • 43
0 votes
1 answer
26 views

Find event date given the probabilities of finding an event

I have a set of clinical notes with dates for each patient and an NLP models which gives a score between 0.0 and 1.0 of a certain event being present in the note. Given the scores, what is the best ...
rhn89's user avatar
  • 101
0 votes
0 answers
10 views

Appropriateness of the Universal Sentence Encoder model

I have a classification problem where the goal is to predict, based on a small paragraph, if an individual is British or not. The model used for the classification is Universal Sentence Encoder (to ...
Sara Mun's user avatar
0 votes
1 answer
33 views

Clustering of large text datasets with unknown number of clusters

I have a list of hotel names which may or may not be correct, and with different spellings (such as '&' instead of 'and'). I want to use clustering in order to group the hotels with different ...
user480840's user avatar
1 vote
0 answers
18 views

BERT eval loss increase while performance metrics also increase

I want to fine-tune BERT for Named Entity Recognition (NER). However, when fine-tuning over several epochs on different datasets I get a weird behaviour where the training loss decreases, eval loss ...
CodingSquirrel's user avatar
0 votes
0 answers
97 views

Locality sensitive hashing (LSH) with word embeddings and cosine similarity

I would like to ask about the methodology of LSH algorithm with Word Embeddings and Cosine Similarity to identify similar documents. First, I tokenize my sentences to create a list of tokens. Then, I ...
BDEngineer's user avatar
0 votes
0 answers
9 views

Problems in understanding Word2vec architectures

I have probably a very simple question, but I did not find any clear resource on the web. First let's consider the Skip-gram model, in which we try to predict a context word given the target word. In ...
user405969's user avatar
2 votes
1 answer
140 views

If a document set is too small for running a topic model, can you simply multiply the document set by a factor of 10 to be able to run the model?

Say I'm using Top2Vec as a topic model to capture the top 10 salient topics across documents. I have an array that contains the documents of the corpus. Initially, there are not enough documents to ...
NominalSystems's user avatar
0 votes
0 answers
73 views

How is the unigram tokenization using EM algorithm?

I intuitively understand what is happening in the unigram tokenizer and I think I also understand the EM algorithm if I can figure out the formulation in which I understand it i.e. What is the latent ...
figs_and_nuts's user avatar
0 votes
0 answers
12 views

Which model to use for word tokenization?

What type of model for word tokenization in a group of words is good?? I want to tokenize skills in a group. There are multiple groups, and I want to predict group number using a single skill. I have ...
bunny's user avatar
  • 1
0 votes
0 answers
10 views

Importance of language models when given simple instruction to a Robot

Do I really need a transformer-based language model if I want to instruct a robot very simple task? Suppose I give an instruction like "Go to the right and pick up the book from the table". ...
Encipher's user avatar
  • 175
0 votes
0 answers
228 views

Bert Used for generative AI

I have a doubt regarding using "Bert" as a generative model. I know Bert can be used for classification or fine-tuning the question-answering. However, is it possible to use Bert to generate ...
Encipher's user avatar
  • 175
0 votes
0 answers
24 views

Evaluation metrics for chunking and synthesis steps for Q&A system

I am doing research and interested in the following question: What are the evaluation metrics for chunking, retrieval and synthesis steps for Q&A, when I do NLQ with LLMs? I am looking for ...
Anakin Skywalker's user avatar
2 votes
0 answers
22 views

Encoder-decoder Transformer model makes outputs predictions almost perfectly but fails to autoregressively decode

The model's sample predictions that I'm printing during training are almost perfect but the model generates meaningless tokens during evaluation. For training I'm feeding it the source and target ...
Sean's user avatar
  • 4,087
0 votes
0 answers
30 views

Identifying figure of speech in the sentence provided machine learning project

Natural Language Processing Figures of speech classification Is it possible to identify the figure of speech in the sentence/s provided as inputs using Natural Language Processing Machine Learning ...
Prashant Akerkar's user avatar
0 votes
0 answers
27 views

Applying Xavier initialization to model causes model to not train properly and therefore output meaningless text

I have a Transformer implementation that I'm working on. The Transformer model is the original encoder-decoder sequence2sequence model introduced in the 2017 paper. I wasn't originally using any form ...
Sean's user avatar
  • 4,087
0 votes
0 answers
108 views

SHAP for text input: aggregating values across instances

I am working on a binary text classification task, where the input is text (~100 words long, but length varies). I am using a fine-tuned BERT-based model. My goal is to get insights about which ...
planetx's user avatar
0 votes
0 answers
69 views

Regression with text data

My goal is to create a regression model with text data where encoded text predicts a value, (news headlines, or article summaries, predicting number of clicks). The y is very left-skewed (few articles ...
user3722736's user avatar
0 votes
0 answers
35 views

Classification in BERT - why not use class as a feature?

I am currently following this post, which details how BERT was trained. I had a few questions about the classification task: In the post, it mentions that the authors of BERT decided to add ...
Victor M's user avatar
  • 319
0 votes
0 answers
10 views

How does Kneser Ney estimate ngrams with BOS without dividing by zero?

The recursive formula for (unmodified) Kneser Ney smoothing is (per Jurafsky08 3.40) $P_{KN}(w_i | h) = \frac{\text{max}(c_{KN}(h\ w_i) - d, 0)}{\sum_v c_{KN}(h \ v )} + \lambda(h) P_{KN}(w_i | h)$ ...
user2740's user avatar
  • 1,386
0 votes
0 answers
39 views

Training an Image Captioning Model with variable number of captions per image

I am following this guide for training an Image Captioning model It uses a dataset which always has 5 captions per image. My dataset greatly varies how many captions I have per image from 1-42. This ...
TwoRice's user avatar
  • 163
2 votes
0 answers
49 views

Can I calculate the significance of the number of deponent verbs with a certain feature like this?

In a language like Ancient Greek, verbal forms are marked for voice (active/middle/passive). Deponent verbs are verbs that exist only in the middle (or passive) voice, but appear to have an active ...
Keelan's user avatar
  • 143
2 votes
1 answer
59 views

Gradient Clipping of Vanilla RNNs vs LSTMs

I am doing an online course that states that the reason we use LSTMs and similar variations of vanilla RNNs is because of the vanishing/exploding gradients problems with vanilla RNNs. However, an ...
HelloWorld's user avatar
0 votes
1 answer
54 views

Why is the WordPiece algorithm implemented according to the maximum mutual information?

WordPiece is a subword segmentation algorithm in the field of natural language processing. Different from BPE, WordPiece will select a pair with the largest mutual information to merge each time, and ...
korangar leo's user avatar
1 vote
0 answers
41 views

Does skipgram model uses backpropagation?

I just started to get interested in natural language processing and I was trying to understand the skipgram model from word2vec. I was reading this interesting website. However, in the mentioned ...
edamondo's user avatar
  • 111
3 votes
3 answers
506 views

Countering t-test "any feature is significant" results for large sample size datasets

I'm doing some analysis over natural language data, which basically entails: Computing some feature over all samples. Evaluating if this feature statistically significantly discriminates between ...
Andre Ye's user avatar
0 votes
0 answers
27 views

How is metadata represented in sentiment analysis?

There are papers on semantic analysis using metadata such as "Sentiment Classification on Steam Reviews" (https://cs229.stanford.edu/proj2017/final-reports/5244171.pdf) and "Detecting ...
soravoid's user avatar
0 votes
0 answers
35 views

Implementation of spBLEU

I was looking for a way to compute statistics for evaluation metrics for language translation models and I came across spBLEU. I can’t find any implementations/examples that would help me start. Does ...
Prithvi's user avatar
  • 21
0 votes
0 answers
83 views

How can BERT/Transformer models accept input batches of different sizes?

I understand that all inputs in a batch need to be of the same size. However, it seems BERT/Transformers models can accept batches with different sizes as input. How is that possible? I thought we ...
The Wanderer's user avatar
0 votes
1 answer
835 views

What is the Llama2 number of steps? [closed]

Llama2 is pretrained with 2 trillion of tokens: $2\times10^9$, and its batch size is of $4\times 10^6$. We can calculate the number of steps (times we upgrade the parameters) per epoch as follows: $$\...
Noether's user avatar
2 votes
0 answers
39 views

Does it make sense to perform Domain Adaptation before Transfer Learning?

Suppose I would like to do extractive question answering on scientific literature. I'm interested in using BERT which was pretrained on Wiki and Bookcorpus. I see two routes here: 1. Fine-tune BERT on ...
Jose Garcia's user avatar
0 votes
0 answers
14 views

Would it be wise to feed k-means results into a cmeans?

I am conducting a cluster analysis of documents using document embeddings as the input to the algorithms. One of the problems I am coming across is that in reality there are documents that belong to ...
osckt's user avatar
  • 31
0 votes
0 answers
35 views

What are more sophisticated measures for word similarity than cosine similarity?

I am trying to compute measures of word/sentence similarity from word embeddings that I would like to use for classification. However, cosine similarity often runs counter to how I as a human would ...
dufei's user avatar
  • 51
0 votes
0 answers
28 views

How does Doc2Vec handle documents where the length is less than the window size?

I am using doc2vec to vectorise documents that average in length 10 words (PV-DBOW implementation of the algorithm). I am wondering how doc2vec handles cases where the number of words in a document is ...
osckt's user avatar
  • 31
0 votes
1 answer
32 views

Considering weights right of the embeddings layer aren't used in Doc2Vec/Word2Vec, is the informative capacity of the embeddings not strongly reduced?

In an extreme (and probably impossible) example, could you not end up with all the power for the prediction being contained in the weights to the right of the embeddings layer?...and thus the matrix ...
osckt's user avatar
  • 31
1 vote
1 answer
99 views

How are vector values assigned initially in Word2Vec and how are they changed with iterations of the algorithm?

I am new to NLP and I'm not fully grasping how word2vec works. I understand that it aims to predict a word given its context or a context given a word but I'm not sure how the initial vector values ...
osckt's user avatar
  • 31
0 votes
1 answer
465 views

Using Word Embeddings in Clustering and Topic Modelling

I am new to the field of NLP and would appreciate any guidance please. I am trying to understand how word embeddings can be used in clustering and topic modelling. If I create word embeddings for ...
osckt's user avatar
  • 31
1 vote
1 answer
185 views

How does training word embeddings bring similar words closer together?

How does training of word embeddings lead to the clustering of similar words in the embedding space? What causes that effect?
Glue's user avatar
  • 485
1 vote
0 answers
197 views

How to determine EC2 instance type and memory for LLM inference endpoint [closed]

I am trying to estimate the costs required for hosting a fine tuned large language model for real time inference. There will be 100s of users querying the endpoint concurrently for multiple use cases ...
user3711946's user avatar
1 vote
0 answers
46 views

Machine learning and Natural Language Processing Algorithms for Indian Surnames Homophones [closed]

Homophones Indian Surnames List English last names Can machine learning, Natural Language Processing (NLP), Artificial intelligence assist in classifying , interpreting and specifying the differences ...
Prashant Akerkar's user avatar
0 votes
1 answer
32 views

Creating a morphology tagging/labeling model

I had an idea of building a model using machine learning or deep learning in order to perform morphological tagging/labeling on untagged/unlabeled data. I have a lot of tagged/labeled data (about 30,...
Dolev Mitz's user avatar
1 vote
2 answers
97 views

Is concatenating a single integer sufficient for encoding positional embeddings in transformer models?

In transformer models, positional embeddings are commonly used to encode the positional information of words in a sequence. While sinusoidal positional embeddings are often employed, I'm curious about ...
Glue's user avatar
  • 485
1 vote
1 answer
60 views

Input non-sequential data of arbitrary size to network

I have a case where I want to feed a network with polylines of data. The problem is that the input can be any number of polylines and the polylines can consist of any number of points. If we instead ...
JakobVinkas's user avatar
3 votes
0 answers
222 views

Attention is All You Need: How to calculate params number of the models?

I want to re-calculate the last column of Table 3 of Attention is All You Need, i.e. number of params in the models. But numbers from my calculation do not match. Model Params from Table 3 ($\times ...
Judd's user avatar
  • 31

1
2 3 4 5
24