Questions tagged [word-embeddings]

Word embedding is the collective name for a set of language modeling and feature learning techniques in NLP where words are mapped to vectors of real numbers in a low dimensional space, relative to the vocabulary size.

Filter by
Sorted by
Tagged with
1 vote
1 answer
25 views

Text similarity for badly written text

Consider the following scenario: Suppose two lists of words $L_{1}$ and $L_{2}$ are given. $L_{1}$ contains just bad-written phrases (like 'age' instead of '4ge' or 'blwe' instead of 'blue' etc.). On ...
user avatar
0 votes
0 answers
17 views

embedding sequences (eg. with doc2vec): do trajectories in the learned (N-dimensional embedding) vector space make any sense?

My data is a collection of temporal sequences of events (each event has a timestamp / or, alternatively, a duration). An example could be sequences of events generated by customers interacting with ...
user avatar
1 vote
1 answer
27 views

Should I use cosine or dot similarity inside word2vec's neural network?

I've implemented the word2vec algorithm according to its negative sampling architecture,using a shallow neural network that performs binary classification on word-embedding vector pairs. The network ...
user avatar
1 vote
0 answers
12 views

How is an embedding space optimized with respect to the loss function?

I understand that the point of the embedding layer is to reduce the dimensionality of the input space while also projecting it onto a space that represents the similarity between the medium in ...
user avatar
  • 619
0 votes
1 answer
37 views

Feedforward Neural Net Language Model - computational complexity (word2vec)

While reading this paper on word2vec, I came around the following description of a feedforward Neural Net Language model (NNLM): It consists of input, projection, hidden and output layers. At the ...
user avatar
0 votes
0 answers
11 views

What is considered as small or large dataset for word2vec?

In the paper "Distributed Representations of Words and Phrases and their Compositionality" that introduced negative sampling (among other things), authors describe the recommended number of ...
user avatar
0 votes
0 answers
8 views

Why are surrounding token embeddings not considered in (simple) NER models?

I just had a look at the hugggingface implemenation of DistilbertForTokenClassification. The token classifier uses the embeddings as input and consists of only one dropout layer and a linear layer. ...
user avatar
0 votes
0 answers
11 views

Word embedding to support OOV words for identity embedding

I want to create a model that performs a user-id embedding (hash of a user) for a Graph Neural Network learning task, the problem I am facing is that I might have a very large corpus of users, which ...
user avatar
0 votes
0 answers
50 views

Using BERT Embeddings + Standard ML for text classification

I am trying to automatically detect whether a text is written by a Machine or a Human. My first approach was using a TF-IDF to build features for a logistic regression classifier, where I got an ...
user avatar
0 votes
0 answers
5 views

Retrieval-based chatbot model - how to minimize distance between context and selected dialogue line?

I'm working on a chatbot system and I'm looking for a way to optimize the distance between the embedded context and the selected dialogue reply. Do you have any paper on the topic, or a possible ...
user avatar
  • 137
1 vote
0 answers
18 views

What is a word embedding approach that would work for these pre-labeled documents?

My Situation: I should start off with my end goal: I want to get a distance metric between each document and all of the other documents To get there, I first need to encode these topic labels so that ...
user avatar
1 vote
0 answers
104 views

fine tune universal-sentence-encoder embeddings

I am new to NLP and Neural Networks. I want to do topic analysis for a dataset of reviews of our product. I tried to use the universal-sentence-encoder along with <...
user avatar
0 votes
0 answers
40 views

What pretrained word embeddings does the Universal sentence encoder use for Deep Averaging Network?

The paper for Universal sentence encoder Universal sentence encoder paper! is pretty straightforward, and so is the paper for Deep averaging network Deep averaging network paper! but I'm confused ...
user avatar
  • 13
0 votes
0 answers
27 views

Continuous Bag of Words NY Time Corpus

I am working to implement the continuous bag of words approach on the New York Times corpus dataset. However, I am getting word embeddings that do not seem very useful based on a few examples of ...
user avatar
0 votes
0 answers
27 views

Given embedding vector A and vector B, how to find top k embedding vectors such that they are similar to vector A and dissimilar to vector B

Which would be better approach for getting top k embedding vectors such that they are similar to embedding vector A and dissimilar to vector B. Approach 1: calculate ...
user avatar
0 votes
0 answers
16 views

Categorical variable converted to embeddings using Embedding layer visual depiction

Say I have a dataset with 3 features 1) Date 2) dayOfWeek with values Sunday to Saturday 3) Number_of_customers If I use One-hot-encoding to convert "dayOfWeek" feature to numeric ...
user avatar
  • 1
0 votes
0 answers
15 views

An efficient way to encode & embed tabular data of a video into a transformer?

So a little bit of a background: I have a folder which contains video files of lets say humans doing a certain action (i.e. walking) where each .2 seconds is documented in a ...
user avatar
  • 111
1 vote
0 answers
13 views

Why use a hash bucket to handle Out-of-vocabulary tokens in embedding layers?

For example, in the nnlm-en-dim128 model in thug (https://tfhub.dev/google/nnlm-en-dim128/1). It says Small fraction of the least frequent tokens and embeddings (~2.5%) are replaced by hash buckets. ...
user avatar
  • 121
0 votes
0 answers
18 views

How to obtain vector representation of phrases in natural language processing and do PCA with it

I am trying to understand from both a conceptual and a Python code point of view, how to represent phrases that are present in a corpus (that is used to train a neural network to classify phrases) as ...
user avatar
0 votes
0 answers
21 views

How to use unit embedding for categorical encoding

I'm trying to improve my model's performance- I am using a kaggle dataset that can be found here. The first columns of the dataset are categorical and look like this: ...
user avatar
  • 101
0 votes
0 answers
41 views

How to train a custom embedding?

I have data with a lot of categorical features. The cardinality of some of these features is quite big (>100), so I want to avoid using one-hot encoding. The idea is to use an embedding. The ...
user avatar
  • 283
2 votes
1 answer
214 views

How does torch.nn.Embedding or tf.keras.layers.Embedding compare to a dense layer?

Embedding layers are a common choice to map some high-dimensional, discrete input to real-valued (computationally represented using floating point) numbers in a much smaller number of dimensions. Some ...
user avatar
  • 78.2k
0 votes
0 answers
45 views

Plotting cosine similarities in 3 dimensional space made from word embeddings

I'm working on a project in which I want to estimate biases from a large corpus of newspaper articles using word2vec. Following this and this paper, biases are calculated as follows. First, a ...
user avatar
2 votes
2 answers
186 views

Word Embeddings with nn.Embedding except glove

I'm not an expert in nlp but I know we usually use glove, word2vec or fasttext ect to get embedding vectors so what is nn.Embedding and what does it do? I mean shouldn't it be specialized for each ...
user avatar
1 vote
1 answer
367 views

Difference between token embedding and character embedding in ELMo model

I am learning about a famous NLP model called ELMo. In the explanations, they talk about two types of embeddings. 1) character representations 2) token representations. Why is there a need to consider ...
user avatar
  • 365
1 vote
0 answers
28 views

Single Embedding Layer for Multiple Categorical columns?

I have 100 binary categorical columns to train a neural network model. Each row will be a vector-like [1,1,0,...1] of length 100. I fed this vector to a neural network for a classification problem and ...
user avatar
  • 53
1 vote
1 answer
27 views

What makes training time longer with bigger parameter size in a deep learning model?

I try to understand, is it always the case with the more parameter you trained, the more training time you need when training a deep learning model. For example, i have a CNN model for text ...
user avatar
1 vote
1 answer
120 views

Understanding number of rnn units in RNN networks

I am trying to learn about recurrent neural networks from here. There are rnn_units = 1024 in the model and each batch contains ...
user avatar
  • 365
0 votes
1 answer
29 views

Ensure trained word embeddings get high similarity with particular words

I am trying out my hand at training a Word2Vec model using gensim. I made a simple training file that basically had just one line repeated multiple times ...
user avatar
1 vote
1 answer
198 views

Does not 'removing the stopwords' affect Natural Language Processing Results in a high degree?

Most stopword lists contain contradicting prepositions (before-after, into-out of) and negativity words (not, no). Removing such words from the text almost always changes the meaning drastically. The ...
user avatar
  • 129
1 vote
0 answers
32 views

PCA: removing dominant vector "directions" (isotropy)

I am currently reading an NLP paper on improving the representation of word vectors in space. The authors show that embedded words are not uniformly distributed in space but are contained in a lower-...
user avatar
1 vote
0 answers
45 views

Definition of the word "embedding"

The mathematical definition of the word "embedding" requires the mapping to be injective, so in that context one speaks of, for example, embedding real numbers in complex numbers (ie, ...
user avatar
  • 796
0 votes
0 answers
14 views

Word vectors projection and residual features

I am working on a project that uses word vectors from word2vec. I can come up with semantic feature vectors by subtracting pairs of word vectors, for example I can say a gender vector is formed by ...
user avatar
  • 395
3 votes
1 answer
132 views

How are sentences one-hot encoded internally in an Embedding Keras Layer?

Multiple references are clear on how a single word is one-hot encoded in an Embedding layer, but what about sentences? In order to illustrate an example, I will use the following SO reference. Let's ...
user avatar
0 votes
1 answer
113 views

Word2vec/SkipGram: Why softmax?

In Word2Vec (SkipGram version), there is a softmax layer at the end of the neural net. As this is expansive to calculate, some approximations are used instead, such as negative sampling. But if in the ...
user avatar
  • 3
0 votes
0 answers
23 views

Word-embedding does not provide expected relations between words

I am trying to train a word embedding to a list of repeated sentences where only the subject changes. I expected that the generated vectors corresponding the subjects provide a strong correlation ...
user avatar
1 vote
1 answer
124 views

How does Word2Vec CBOW softmax work with multiple context words?

I'm referring to following paper from Xin Rong - "word2vec Parameter Learning Explained", to be precise the equation (4): $$ p(w_j|w_I) = \frac{\exp(\mathbf{v’}^{T}_{w_{j}}\mathbf{v}_{w_{I}})...
user avatar
  • 113
1 vote
0 answers
108 views

Rule of thumb for the minimum frequency for unknown words in a NLP Neural Network Language model?

I know there are approaches that process unknown words with their own embedding or process the unknown embedding with their own character neural model (e.g. char RNN or chat transformer). However, ...
user avatar
0 votes
0 answers
28 views

Cross entropy error: Poor modelling giving too much weight to unlikely events

I was reading this paper. link (page 5) In this paper, there is a statement that goes like this: To begin, cross entropy error is just one among many possible distance measures between probability ...
user avatar
  • 279
1 vote
1 answer
36 views

Generating embeddings for languages without a written representation?

I'm considering the topic of generating an NLP Embedding for a language without a written standard or a significant corpus. I realized that as challenging as that is, it is still not as challenging as ...
user avatar
  • 10.7k
1 vote
0 answers
8 views

The embeddings of Google document is stemmed or not? [closed]

I will use the embedding google document to do some project. Just want to know whether it was stemmed of the words?
user avatar
1 vote
0 answers
47 views

Learn sentence embeddings from a sequence of token embeddings

I want to build a sentence classifier that takes the sentence as a sequence of token embeddings. I'm specifically interested in the methodology for learning the sentence embedding from the sequence of ...
user avatar
  • 111
1 vote
1 answer
119 views

Extracting word embedding features of a sentence using Transformer-XL

As you know, there are several pre-trained models that we can use to extract word embeddings. As an example, I can use the following codes to retrieve word2vec features of my text: ...
user avatar
  • 365
1 vote
0 answers
40 views

Skipgram model theory confusion

In the output layer of a skipgram model, there are $|\text{Context}|*|\text{Vocab}|$ values. And for each context word, the values are basically the dot product of the input word representation and ...
user avatar
1 vote
1 answer
631 views

Distance measure in word2vec

I am currently learning about word embedding and word2vec, and I am having a hard time understanding how the similarity between words is measured in that representation. I have often read that the ...
user avatar
  • 121
1 vote
1 answer
133 views

Intuition for GloVe word embeddings

I am currently looking at the formulation for the GloVe word embedding model. I have a difficult time understanding the intuition behind why the ratio of co-occurence probabilities are used. The ...
user avatar
  • 886
1 vote
2 answers
167 views

Is there a seq2seq model that can encode sentences that include numerical values?

I am trying to build a seq2seq model that encodes sentences which include numerical values. For example, Patient's systolic blood pressure was 128. Conventional ...
user avatar
  • 11
0 votes
0 answers
19 views

Choosing a model for input: categorised, weighted sequence, output: binary variable

What would be an appropriate model for predicting a binary target variable, given a weighted sequence? Sequences will be reasonably short, typically between ~ 1 and 5 elements. I have in the order of ...
user avatar
  • 101
1 vote
1 answer
150 views

What's the best practice for dealing with OOV characters?

I have read on the advantages of using character-level language models over word-level ones. In particular, you don't have to deal with the problem of out of vocabulary (OOV) words, since characters ...
user avatar
4 votes
1 answer
56 views

Why do we not use continuously defined losses in NLP?

I understand that various problems in optimization in NLP which do not exist on continuous tasks such as vision, arise in NLP because we do not have continuous data to predict, but one-hot vectors ...
user avatar
  • 43

1
2 3 4 5