Natural Language Processing is a set of techniques from linguistics, artificial intelligence, machine learning and statistics that aim at processing and understanding human languages.

learn more… | top users | synonyms (1)

0
votes
0answers
9 views

Difference between dynamic pooling and static pooling in convolutional neural networks

Since yesterday I was thinking that pooling layer in CNN has fixed size(e.g. 2 by 2). Then I saw in this paper: http://phd.nal.co/papers/Kalchbrenner_DCNN_ACL14 We define a convolutional neural ...
0
votes
0answers
15 views

Calculate statistical significance in natural language processing

I have a task to say whether the difference in performance between two systems is statistically significant. The task is similar to sentiment analysis. I have sentences and I need to classify them ...
0
votes
0answers
10 views

Almost a multinomial logistic regression

Is there a way to train a multinomial logistic model where the true classification is unknown, but summary information obtained from the true classifications is known? Illustrative Example: 5 ...
0
votes
0answers
10 views

Latent Dirichlet allocation: how to derive $\theta$ given $\alpha$?

I have been studying Latent Dirichlet allocation (LDA) since quite long. I have a confusion in Dirichlet priors. For example, if I consider 3 topics and take $\alpha_1 = 0.1$, $\alpha_2 = 0.1$ and ...
0
votes
0answers
34 views

Build HMM of text data in R

I'm trying to make my own HMM tagging in R but don't know how to estimate parameter values since the packages I have been working with haven't worked with my data. The latest package I have been ...
1
vote
0answers
22 views

How does Word2Vec's skip-gram model generate the output vectors?

I am having problems understanding the skip-gram model of the Word2Vec algorithm. In continuous bag-of-words is easy to see how the context words can "fit" in the Neural Network, since you basically ...
1
vote
0answers
29 views

calculating PMI for co-occurrences of words

I am in the process of building a question answering system. I am interested in calculating the PMI for words $x$ and $y$ occurring within 5 words of each other in a document. I have the formula and ...
1
vote
1answer
59 views

In word2vec, for analogies do we use “in” or “out” vectors?

In word2vec each word is associated with two vectors (one for in and one for out) so that it predicts conditional probability: $$P(word_{out}|word_{in}) = \frac{\exp(v_{in} \cdot ...
0
votes
0answers
6 views

NLP to correlate two statements

What’s the best mechanism of relating / co-relating two distinct sentences to be on the same subject / context ? If a sentiment analysis program read 2 statements as :- I went to a movie The ...
0
votes
0answers
11 views

Numerical POS tag training in nltk (python)

This is a follow-up question to my previous question posted here. To create a natural language calculator, I tried TrigramTagger from nltk. For training, I want to tag multiplication and 2 numbers. ...
0
votes
0answers
18 views

Classification with restrictions

I am working with multi-class classification. I have two sources of information for my classifier: I can get information only from the sample $x_i$. So my analyzer produces quite big number (~600) ...
0
votes
0answers
48 views

Natural language calculator using machine learning

What machine learning algorithm should I use here (python)? I wish to create a natural language maths calculator. To start, I will focus only on multiplication of 2 numbers. For example: for all ...
1
vote
0answers
16 views

classification real vs. made-up words

I am interesting in building a classifier that can separates made-up words (such as brands) from real words (belonging to the English dictionary for example). I have tried using a Soundex ...
0
votes
0answers
9 views

When is a perplexity score suboptimal?

I'm doing an NLP project where I calculate the perplexity of N sentences. And so though not entirely relevant to the project it inspired the following question: When is a perplexity good, suboptimal ...
1
vote
1answer
67 views

Recursive neural network implementation in Theano

Is there any available recursive neural network implementation in Theano? Theano's deeplearning.net tutorial does not present any recursive neural networks. Most Theano code I've found is CNN, LSTM, ...
3
votes
2answers
58 views

Visualize a lot of words

I have a CSV file with a set of words occurrences in several documents. The first column is the document it. The second column states the text topic (there are 5 different topics). The other columns ...
3
votes
1answer
39 views

Mixed effects model - Assumptions, Comparison

I only have a very basic understanding of statistics, but I want to see if the variables consonant, vowel, ...
0
votes
0answers
12 views

Do “bad training instances” decrease predictions quality in multi-label text classification with SGD?

I have 150k company descriptions (~140 characters long) tagged with approximately 1-6 industries. I have 110 possible industries. Industry distribution across different companies is not homogeneous: ...
0
votes
0answers
14 views

n-gram language model

At the end of the introduction of A Neural Probabilistic Language Model (Bengio et al. 2003), the following example is given: Having seen the sentence ...
1
vote
1answer
36 views

Curse of dimensionality with language models

In the seminal paper A Neural Probabilistic Language Model, Yoshua Bengio and his colleagues make the following point: If one wants to model the joint probability distribution of 10 consecutive ...
0
votes
0answers
17 views

Where can I find an implementation of GPU friendly sparse linear batches for torch?

I am training a language model, the size of the vocabulary is about 420k unique tokens. The input and output representations are 1-hot vectors. I'm planning to use a sparse linear input layer and a ...
0
votes
0answers
15 views

Why word2vector introduct an aid vector for computation?

I am reading the word2vec code by Miklov. In the Skip-gram, negative sampling method, the word vector is syn0. But an aid word vector is introduced called syn1neg, I want to know why we don't operate ...
6
votes
1answer
113 views

Deep Learning - What are the senones in a Deep Network?

I am reading this paper: skype translator where they use CD-DNN-HMMs (Context dependent Deep neural Networks with Hidden Markov Models). I can understand the idea of the project and the architecture ...
1
vote
2answers
36 views

Classification of a person's name versus a company name

My current problem is that I have a long list of strings that need to be classified as either a "Person", "Firm", or "Bad Data". Some of these were human entered, so there are a bunch of misspellings ...
0
votes
1answer
45 views

Is a word embedding a vector or a function?

Is a word embedding a vector or a function? I have read contradicting statements: Function: A word embedding $W: \text{words} \rightarrow R^n$ is is a paramaterized function mapping words in some ...
0
votes
0answers
13 views

Could i combine character and word n-gram features on a single feature space?

I'm working on a text classification problem using n-gram language models, I have both separate models (character-based and word-based) but I wanna combine them and I'm wondering if that's possible.
0
votes
1answer
14 views

Method for inferring tree structure from messy sequence data

I have data about case events in criminal cases. These data are in the form of sequence like ...
2
votes
0answers
24 views

LSA projections of documents and terms

I am trying to understand how Latent Semantic Analysis works, reading demonstrations based on singular value decomposition. Let's denote $X$ a $D \times W$ document-term matrix. The $D$ rows of $X$ ...
1
vote
0answers
61 views

Why is skip-gram better for infrequent words than CBOW?

I wonder why skip-gram is better for infrequent words than CBOW in word2vec. I have read the claim on https://code.google.com/p/word2vec/.
0
votes
0answers
10 views

Is the comparison I'm making between these two language corpora valid/appropriate?

I having been analyzing presidential speeches for informal language. I have a repository of ~900 speeches, and ~3.5 million words. The measure of "informal language" I chose was how often the ...
2
votes
1answer
39 views

How many words does the algorithm search through in Google Ngram? [closed]

When I run a query for "hers" in Google Ngram Viewer, I get back the word's frequency of occurrence as a percentage. We know the outcome percentage; what's the denominator on the other size? Is it 100 ...
1
vote
1answer
190 views

Why is hierarchical softmax better for infrequent words, while negative sampling is better for frequent words?

I wonder why hierarchical softmax is better for infrequent words, while negative sampling is better for frequent words, in word2vec's CBOW and skip-gram models. I have read the claim on ...
1
vote
0answers
19 views

What are the pros and cons of applying pointwise mutual information on a word cooccurrence matrix before SVD?

One way to generate word embeddings is as follows: Get a corpora, e.g. "I enjoy flying. I like NLP. I like deep learning." Build the word cooccurrence matrix from it: Perform SVD on $X$, and ...
0
votes
0answers
38 views

Difference between pointwise mutual information and log likelihood ratio

I know this is a very silly question. But i came across some papers on statistical methods in natural language processing, particularly Ted Dunning's paper, and there i found the formula that he has ...
0
votes
0answers
11 views

What is a “word type”?

I have seen this term referred to when reading papers about linear models, but am not positive what the meaning is. My guess is that a "word type" is simply a unique word, or that "word type" somehow ...
2
votes
1answer
72 views

multi-class classification with word2vec

My problem: The input data is a corpus of short documents (a few sentences each). In each document some expressions need to be classified to categories. A document must contain some categories (each ...
4
votes
0answers
100 views

Should I normalize word2vec's word vectors before using them?

After training word vectors with word2vec, is it better to normalize them before using them for some downstream applications? I.e what are the pros/cons of normalizing them?
2
votes
1answer
128 views

Input vector representation vs output vector representation in word2vec

In word2vec's CBOW and skip-gram models, how does choosing word vectors from $W$ (input word matrix) vs. choosing word vectors from $W'$ (output word matrix) impact the quality of the resulting word ...
3
votes
1answer
41 views

Statistical tests for 3-gram collocations extraction

I'm trying to extract collocations from some text data, and I use statistical tests to tell if an n-gram is a good collocation candidate or not. All the sources I came across so far (for example, ...
3
votes
1answer
42 views

Is there an NLP or text classification method for word order and pre-specification of terms?

I'm fairly new to NLP and text classification world, and so far I haven't been able to find the answer to the following problem: I have text entries for a large number of observations. My goal is to ...
1
vote
0answers
20 views

Expectation-Maximization for NLP tasks

I am looking for resources on Expectation-Maximization for NLP tasks. Ideally, they should be mathematically thorough (vs. just take some soft counts here and there), and give some clear intuitions.
2
votes
0answers
79 views

Is this popular approach to Bootstrap hypothesis testing correct?

A widely cited paper "Statistical Significance Tests for Machine Translation Evaluation" proposes usage of Bootstrap for evaluating significance of difference across systems for Machine Translation ...
0
votes
0answers
55 views

Machine learning: Combining NLP with numerical feature

I have a list of numerical / logical features which I want to use for supervised learning (e.g. random forest > product info like price, color, etc. to predict rating). I also have a lot of ...
1
vote
0answers
60 views

how to use CRF package in R [closed]

How can I use the CRF package in R to create a part of speech tagger? I need sample input training data sample code to train the CRF model
2
votes
1answer
28 views

KDD 15 paper: scoring bigrams

I am by no mean into Statistics nor NLP. And I am reading the following paper, trying to learn something new: ...
0
votes
1answer
57 views

Linear regression with categorical independent and dependent variables?

Can linear regression be used when both the dependent and independent variable are categorical? i am looking at word-frequency distribution among a series of texts, and want to show that there is a ...
2
votes
0answers
17 views

Inter-annotator agreement in multi-label classification

I wonder what is the best metric to measure inter-annotator agreement in multi-label classification for two annotators?
0
votes
0answers
38 views

Cramer's V and word frequency

I am investigating the frequency of a word in two distinct corpora. In one corpus, it occurs 232 out of 189,489 times, and in the other corpus it occurs 4394 times out of 559,715 times. The result of ...
6
votes
0answers
78 views

Word embedding algorithms in terms of performance

I'm trying to embed roughly 60 million phrases into a vector space, then calculate the cosine similarity between them. I've been using sklearn's CountVectorizer ...
0
votes
0answers
47 views

Log vs. double normalizing of Term Frequency

In the wiki page on Term Frequency-Inverse Document Frequency (tf-idf) there are two interesting TF weights: Double normalization $K$: $K + (1-K)\frac{f_{t,d}}{\max_t f_{t,d}}$. In this question I ...