Questions tagged [natural-language]

Natural Language Processing is a set of techniques from linguistics, artificial intelligence, machine learning and statistics that aim at processing and understanding human languages.

Filter by
Sorted by
Tagged with
0
votes
0answers
5 views

WHICH weight matrix are shared in RNN and which change upon Time?

In my opinion there are basically 4 weight matrix in the RNN. SO there are different names given in different scenarios but let me point out what I know about RNN in very simple terms. Please correct ...
0
votes
0answers
11 views

Validity and Meaning of Feature Importance in Neural Networks

I’m trying to identify the features of products which consumers are most passionate about through sentiment analysis of product reviews. While there are plenty of models which seem to be quite ...
0
votes
0answers
21 views

Adam optimizer in sPacy giving strange results

I have the following toy example to play with the TextCategorizer CNN model in sPacy. Given the distribution of my input dataset, I am expecting the test results to ...
0
votes
0answers
8 views

How do the linear layers in the attention mechanism work?

I think I now the answer to my question but I dont really get confirmation. When taking a look at the multi-head-attention block as presented in "Attention Is All You Need" we can see that ...
1
vote
1answer
18 views

Why is the softmax on the dot product of the word embedding is the probability of context given word?

I was learning about the Word2Vec model, and the following equation was shown: $\huge{p(o|c) = \frac{exp(u^T_ov_c)}{\sum_{w\in{V}}exp(u^T_wv_c)}}$ in words, the probability of the context word given ...
0
votes
0answers
11 views

Word2Vec vs. Doc2Vec Word Vectors

I am doing some analysis on document similarity and was also interested in word similarity. I know that doc2vec inherits from word2vec and by default trains using word vectors which we can access. My ...
0
votes
2answers
11 views

Is there a seq2seq model that can encode sentences that include numerical values?

I am trying to build a seq2seq model that encodes sentences which include numerical values. For example, Patient's systolic blood pressure was 128. Conventional ...
0
votes
0answers
22 views

Time dependent representation for time series events with different time gaps?

In natural language processing, we can treat characters as evenly spaced time series in RNN models where time gaps are independent of the sequence and only sequential positions matter. If I want to ...
1
vote
1answer
15 views

How many emails would I need to train a good text extraction model?

I'm looking to train a model that will identify product names in an email that a user has bought. The end result would be something very much like named entity extraction, except this should correctly ...
1
vote
1answer
14 views

Why are language modeling pre-training objectives considered unsupervised?

Maybe this is stemming from my not-so-great grasp of supervised vs. unsupervised learning, but my understanding is that if we have access to ground-truth labels then it's supervised learning and if ...
0
votes
0answers
17 views

NLP: vectorizer/metric to upweight absence of frequent terms

I'm doing hierarchical clustering of documents in a corpus; there are words that occur in almost all the documents. To define document similarity, I've used ...
0
votes
0answers
5 views

Explicitly Marking Word Boundaries for 1D CNNs

I'm doing character embedding for NLP tasks using one-dimesional convolutional neural networks. I haven't found any empirical evidence of whether or not marking the word boundaries makes a difference. ...
0
votes
0answers
6 views

Breaking substitution cipher with language model

Frequency analysis is a common tool used to break substitution ciphers, but often relies on intuition and guesswork of a human. Since language models can objectively calculate perplexity (how ...
1
vote
2answers
26 views

Tweet classification - use of features

I have been given the task to classify some tweets per topic. I have done a classification based only on the per-document-per-topic probability with LDA. I have been suggested to use BTM instead, so I ...
0
votes
0answers
12 views

Use of ignore_index on CrossEntropyLoss() for text models

I have been using PyTorch's CrossEntropyLoss() on a Language Autoencoder. I noticed that most people use ignore_index for ignoring the pad token in loss calculation eg this. From what I understand ...
0
votes
0answers
9 views

Tokenization with charachter offsets for NER

I'm working on NER, and have labeled data with char offsets. Is it possible to somehow finetune a tokenizer so that it'll be compatible with the char offsets? I.e., a token is either inside the offset ...
1
vote
0answers
36 views

Classify tweets per topic

I am approaching machine learning for the first time because of my studies. I have been given a bunch of tweets and the goal is to classify them per topic. I really have no clue on how this should be ...
0
votes
1answer
16 views

RNN Loss in Sentiment Analysis

I am currently reading on RNNs and Backprop through Time. With MLPs using SGD, we did Backprop after every training sample. With RNNs, one method to avoid exploding gradients is to cut an input sample ...
2
votes
1answer
45 views

Shouldn't ROUGE-1 precision be equal to BLEU with w=(1, 0, 0, 0) when brevity penalty is 1?

I am trying to evaluate a NLP model using BLEU and ROUGE. However, I am a bit confused about the difference between those scores. While I am aware that ROUGE is aimed at recall whilst BLEU measures ...
1
vote
0answers
16 views

Language Model dealing with Book Dialogue

My goal is to generate text based on a specific book using an LSTM language model. One problem with my generator is that it seems that the book's dialogue is somewhat messing up my generator. I had ...
4
votes
2answers
56 views

Generating text from language model

I have a trained LSTM language model and want to use it to generate text. The standard approach for this seems to be: Apply softmax function Take a weighted random choice to determine next word This ...
1
vote
0answers
12 views

Decoder's autoregressive output keeps outputting the same token

I'm currently working on a personal reimplementation of the Transformer model from the paper Attention is All You Need (Vaswani et al., 2017) and had a question. Right now the way that I'm getting ...
0
votes
0answers
46 views

Hierarchical softmax vs softmax in hyper-parameter search

I am training an NLP model using fasttext where fasttext allows you to use either hierarchical softmax or softmax. It is my understanding that hierarchichal softmax is orders of magnitude faster ...
0
votes
0answers
9 views

Two Representations in Word2Vec (Skip-Gram) - Neural Network Interpretation

On this platform, it has been discussed [1] why Word2Vec in its original description uses two vectors for each word (one if the word is the middle word, the other if the word is the context word). ...
0
votes
0answers
14 views

How much does upcasing benefit NLP?

I have been wokring on an NLP project at work where the available training data has all been preprocessed (upcased, some characters removed etc.). However, I have just been informed that the data I ...
0
votes
0answers
6 views

Neural Text Simplification with no appropriate dataset

I'm currently starting a research project focused on NLP. One of the steps involved in this project will be the development of a text simplification system, probably using a neural encoder-decoder ...
0
votes
1answer
21 views

What's the best practice for dealing with OOV characters?

I have read on the advantages of using character-level language models over word-level ones. In particular, you don't have to deal with the problem of out of vocabulary (OOV) words, since characters ...
1
vote
0answers
36 views

In the attention mechanism, why are there separate weight matrices for the queries and keys?

To perform self attention over a collection of $n$ vectors stacked up into a matrix $X \in \mathbb{R}^{n \times d}$, we first obtain query, key, and value representations of these vectors via three ...
0
votes
0answers
14 views

Part-of-Speech tagging: what is the difference between known words and unknown words?

I am trying to understand the result evaluation table (table 1) of this paper. There are three different accuracies reported overall, ...
0
votes
1answer
12 views

How does recall help precision overcome “length-related problems?”

I'm reading the paper on the Bilingual Evaluation Understudy (BLEU) metric BLEU: A Method for Automatic Evaluation of Machine Translation (Papineni et al., 2002) and had a question regarding a quote ...
0
votes
0answers
48 views

Is it possible to use Transformer model for binary classification of unbalanced panel data?

So I have unbalanced panel data, i.e. multivariate time series with different lengths for each individual and a binary label at each time point. I was thinking to use some deep learning approach and ...
0
votes
0answers
13 views

CTC Speech Recognition Model giving absurd results on actual recording

I have trained a speech recognition model which uses CTCLoss and is inspired from https://www.assemblyai.com/blog/end-to-end-speech-recognition-pytorch I trained it on the Librispeech Dataset (train-...
0
votes
1answer
30 views

Is statistical machine translation similar in many ways to Hidden Markov Model? How can we justify it?

Came across this question but didn't know the correct answer, if anyone could clear this out would be of great help. 'If we say we Statistical machine translation is similar in many ways to Hidden ...
0
votes
0answers
9 views

Comparison of stemmed and unstemmed word embeddings (W2V)?

I have a corpus of 10,000 documents. Now I create two wordembedding spaces using a W2V model: I first stem all words in the corpus and then train a W2V model on it. I train the W2V model on the ...
1
vote
0answers
16 views

Trained BERT models perform unpredictably on test set

We are training a BERT model for a sequence labeling task with six labels: five labels indicate that a token belongs to a class that is interesting to us, and one label indicated that the token does ...
0
votes
0answers
7 views

How to perform large scale multihierarchical text classification?

There is a dataset from an old kaggle competition, https://www.kaggle.com/c/lshtc/discussion/7980 and I wanted to work on it as I am learning NLP. I have done a ...
0
votes
0answers
16 views

How to evaluate the quality of text data?

I am trying to perform sentiment analysis using some french pretrained models from the Huggingface Library. The initial test aren't great. Most of my sentences are rated as LABEL_1 with a confidence ...
0
votes
0answers
25 views

Topic Modeling: topic coherence score by each topic

I'm currently using NMF, but I am just curious whether I can have coherence scores for each topic. For instance, i want to see whether topic 5 is trained better than topic 1. By looking at the ...
0
votes
1answer
11 views

When to use documents vs. sentences for Word2Vec?

I have a collection of words from different communities. Each community has a different way of using language and will provide a different word embedding. I can concatenate the sentences from the ...
0
votes
1answer
14 views

conceptually, how to use NLP to predict a numeric output

suppose I'm trying to use medical notes to predict the cost of medical service. For example, a patient will call in, tell the operator how they feel, their diagnosis, etc etc, and the operator will ...
0
votes
0answers
13 views

How do define the “spread outness” of entities within a document?

I'm currently working on an NLP project. I want to see whether how spread out entities are in a document would affect downstream performance. The method that I'm currently applying is to check whether ...
4
votes
1answer
39 views

Why do we not use continuously defined losses in NLP?

I understand that various problems in optimization in NLP which do not exist on continuous tasks such as vision, arise in NLP because we do not have continuous data to predict, but one-hot vectors ...
0
votes
0answers
11 views

Estimating the best length of window for winnowing

I have to analyze biographies or descriptions for social media profiles (20-40 words) and compare them to the user input to check if we have found a correct person. What window length is it better to ...
0
votes
0answers
13 views

Interpretation of token weights in LDA topic model?

I'm using Gensim's LDA model for some topic modeling. Once the model has trained, it provides for each topic a coherence score, as well as a list of tuples of weights and tokens. The latter could for ...
0
votes
0answers
16 views

How does stackoverflow find similar questions?

I find that stackoverflow is very good at finding similar questions after I type in my question title. Just curious how to achieve this using NLP techniques. Suppose we have a collection of all ...
0
votes
0answers
37 views

Determine if resume meets requirements of job description

What would be the best approach to determining if, or how much a resume meets the requirements of a job description. I understand you could extract features from both texts with Latent Dirichlet ...
0
votes
0answers
4 views

Open source pre-trained models for taxonomy/general word classification

given any two words I'd like to understand if there's some sort of taxonomy/semantic field based relationship. For example given the words "Dog" and "Cat" I'd like to have a model ...
0
votes
0answers
42 views

Logistic regression vs XGBoost

I am doing a binary classification problem. I found that using TF-IDF(sparse matrix, more than 100,000 dimensions) as input, Logistic regression performs better than XGBoost. Using Doc2vec(100 ...
2
votes
1answer
30 views

How to find the closest table name for a given mis-spelled table name?

Suppose a database has a list of tables. A user can enter a table name to get the table information. If he/she enters a mis-spelled table name, then the system needs to scan all table names and return ...
0
votes
0answers
14 views

Is there a way to provide multiple masks to BERT in MLM task?

I'm facing a situation where I've to fetch probabilities from BERT MLM for multiple words in a single sentence. ...

1
2 3 4 5
20