Statistical analysis of texts expressed in languages spoken or written by people, such as English or Norwegian.

learn more… | top users | synonyms

0
votes
0answers
4 views

Weka - StringtoVector Filter Not working [migrated]

I am practicing Weka using the Reuters data. The StringtoVector Classifier works for converting my string data (shown below), so I can analyze the articles to understand what words predict the ...
2
votes
1answer
20 views

How to handle changing input vector length with neural networks

I want to train a neural network with a sequence of character as an input vector. Learning examples have different length and for this reason I don't know how to represent them. Let's say I have two ...
0
votes
0answers
34 views

Sentiment Analysis - How should I handle negatively biased word list length?

I'm implementing a simple sentiment analysis algorithm where the authors of the paper have a word list for positive and negative words and simply count the number of occurrences of each in the ...
1
vote
0answers
16 views

How to understand the patterns of section names in a resume?

recently I am doing some text mining works with resumes. The objective is to divide the resume into several sections based on its headings and contents and then classify it for required jds. Eg. We ...
0
votes
1answer
33 views

Word-frequency and statistics

I am new to statistics and am wondering how I can apply it in linguistics. There is a conjunction in a corpus that 902 times (.91) conjoins sub-clausal units, and 91 (.09) times clausal units. How ...
0
votes
0answers
12 views

Using LIWC for ANOVA and pairwise analyses

I want to use a tool called Linguistic Inquiry and Word Count (LIWC) to analyze text data. As far as I know, it counts the number of words found in a given text that match a set of dimensions (e.g., ...
2
votes
1answer
28 views

Comparing corpus complexities

I would like to compare how complex (varied or predictable) are my three corpora. They are from different topics, so some vocabulary is different, some is the same. Looking at one of the data sets ...
0
votes
0answers
8 views

How alignment between words are generated by IBM model1?

I implemented the translation model IBM1. As a result, I got the translation table P(targetWord|sourceWord) wich is ok. I want also to obtain the alignment of words in the corpus that I used in order ...
2
votes
3answers
52 views

A good intro to computational linguistics?

I have a pretty good background in data analysis and statistics in the social sciences, including both frequentist and Bayesian paradigms, and I have recently been introduced to computational ...
1
vote
0answers
18 views

How to learn (arbitrary) constraints for selecting the best candidate from a group?

In my classification problem, each instance is a group of possibly hundreds of candidates, from which only one should receive the label $True$ and the remainder the label $False$. For example, in ...
0
votes
1answer
51 views

How to standardize text data for training Neural Networks?

I want to train neural network with text data(natural language) as input for classification purpose. One way for standardizing text data for neural network is to use N-GRAM/SKIP-GRAM representation ...
0
votes
0answers
29 views

How to find the perplexity of a corpus

The formula of the perplexity measure is thus: $ p: \left(\frac{1}{\sqrt[n]{p(w_1^n)}}\right) $ where: $p(w_1^n)$ is: $\prod_{i=1}^n p(w_i)$ If I understand it correctly, this means that I ...
2
votes
1answer
25 views

determining significance of term use

Thing one: feel free to RTFM me: I'm definitely looking for search-able terms or background reading. Our situation is this: we have a set of 140 reviewers and 20 elements. Each reviewer reviews each ...
1
vote
1answer
33 views

Neural language model training - stochastic vs batch

Dealing with a very basic neural language model: 3 words of context, vector size 100, one hidden layer size 200, vocabulary size 1000, predicting the next word with a softmax output layer. Previously ...
2
votes
0answers
16 views

How to measure how 'well' I am matching Google keywords?

For google keywords you can bid on a broad match. For example let's say I bid on the keyword 'best hamburger' and somebody searches 'What sort of beef makes the best hamburger?' and 'eat best ...
0
votes
0answers
27 views

Hashing functions in NLP

I have been reading a lot of papers about nlp which use the hashing trick, and I came across a lot of sentences like : "We take k hashing functions to hash words or bi-grams". And after that they ...
0
votes
1answer
37 views

How do search engines generate related searches?

I would like to know how search engines like Bing generate related searches when the user starts typing into the search box. From what I gather, there has to be some sort of a ranking algorithm where ...
0
votes
1answer
28 views

How can I show that my sample is random and a good representation of the population?

Okay, I'm looking at a population of user reviews. I have collected a random sample of the reviews and studied the trends of the words used and sentence structure. How do I make inferences about ...
0
votes
0answers
55 views

Trying to understand the basics of a mixed-effects logistic regression model for a 10-step continuum

I am trying understand how to correctly build a mixed-effects logistic regression model in R. I believe my model is pretty simple and straight forward but I'm lacking in experience and uncertain I'm ...
1
vote
0answers
114 views

Any advice on how to improve my accuracy rate in text classification?

I'm trying to do a text classification task. Here are some specs: Context file size = 1M+ documents already labeled Number of top-labels = 17 Number of sub-labels = around 130 Each document is ...
0
votes
0answers
17 views

Unsupervised feature learning from raw text as a previous step for clasification?

I have a corpus of 2500 opinions, is it posible to use scikitĀ“s restricted boltzmann machine implementation to extract a feature vector as a previous step to a classification task?. What aproach do i ...
2
votes
2answers
80 views

Weighting words based on position in text

I'm currently working on semantic analysis and had a question about text organization and structure. Are there any algorithms, or statistical / machine-learning models that weight the importance of a ...
1
vote
1answer
109 views

How to handle unseen features in a Naive Bayes classifier?

I am writing a naive bayes classifier for a text classification problem. I have a bunch of words and an associated label: ...
0
votes
1answer
34 views

Feature learning with a deep learning aproach?

How to create a feature vector from text with a deep learning aproach?. Im new at this topic, could anybody advice me where to start and how to aproach this task?.
4
votes
0answers
102 views

Understanding Singular Value Decomposition in the context of LSI

My question is generally on Singular Value Decomposition (SVD), and particularly on Latent Semantic Indexing (LSI). Say, I have $ A_{word \times document} $ that contains frequencies of 5 words for ...
0
votes
0answers
139 views

What algorithms should I use to perform job classification based on resume data?

Note that I am doing everything in R. The problem goes as follow: Basically, I have a list of resumes (CVs). Some candidates will have work experience before and some don't. The goal here is to: ...
1
vote
1answer
82 views

Calculating Perplexity

In the Coursera NLP course , Dan Jurafsky calculates the following perplexity: Operator(1 in 4) Sales(1 in 4) Technical Support(1 in 4) 30,000 names(1 in 120,000 each) He says the Perplexity is 53. ...
2
votes
1answer
40 views

Question about the probability chain rule

I've understood from this: Is this a correct statement of the probability chain rule? that in the chain rule for probability, conditioning can be done on different variables. I was wondering what ...
1
vote
1answer
385 views

Naive Bayes with unbalanced classes

As a part of a project for the university is should train a Naive Bayes classifier to classify question and answers in three different categories, the task should be easy since that the 3 classes are ...
2
votes
1answer
149 views

Visualizing Mutual Information Against TF-IDF for Text Corpus Data

I'm working on a data visualization project for the semester and have decided to work with a corpus of discussion forum data focused around debate over political issues (available here). I'm ...
1
vote
1answer
108 views

Hidden Markov Model: Predict observation sequence from state sequence

Given a transition matrix, starting probability, means and covariances Is it possible to predict the most likely observed sequence for a given state sequence using the above details? If yes, how? ...
0
votes
0answers
34 views

Issue With class imbalances in classification problem

I am working on a word sense disambiguation problem. Specifically, I am using decision lists to classify the ambiguous word. Decision lists work in the following way. The quantity ...
0
votes
0answers
34 views

Word Sense Disambiguation in Practice

I have a question that might seem very obvious but I don't really have a good answer for it. There are many algorithms out there that deal with word sense disambiguation but all of the ones that I ...
1
vote
0answers
60 views

How to extract structured information from a text string?

I have a text string containing unstructured data and I would like to analyze it in order to extract structured information. In particular, this text string specifies when a service is operational ...
0
votes
1answer
96 views

Multiple Bernoulli and Multinomial Distirbution

It's well known that language can be modeled by Multinomial distribution and Multiple Bernoulli distribution. So far I don't see any advantage of Multiple Bernoulli distribution representation over ...
5
votes
0answers
65 views

Expectation Maximization Clarification

I found very helpful tutorial regarding EM algorithm. The example and the picture from the tutorial is simply brilliant. Related question about calculating probabilities how does expectation ...
0
votes
1answer
50 views

Improvement on duplicating instances

I have a task of Relationship extraction. There are some set of predefined relations in the corpus. I need to train classifier to recognize the type of relation or the lack of relation between every ...
1
vote
2answers
195 views

Most important journals in data mining/ML, NLP and IR?

Can you please provide with me with the names of the most important journals in data mining, machine learning, natural language processing and information retrieval?
1
vote
0answers
75 views

Calculating and Normalizing ngram relevancy scores from free text extraction

I currently look for a set of ngrams in many sets of documents to establish a relevancy score for each set - eg. I look for the n-gram "adhesive tape" in ~1M sets of 1-500 documents. The values I ...
2
votes
1answer
222 views

Regression analysis in R using text field?

I'm working in R. I'd like to run a regression analysis for predicting price against terms in a text field. I have a dataset of jewellery auction listings, with price paid, date, and an unstructured ...
1
vote
0answers
22 views

Early papers using statistical models for word completion?

I am trying to find references about (early, as in pre-2000, but i would really love something pre-1990) statistical language models in word completion (like T-9, or Google's search autocomplete), but ...
3
votes
2answers
158 views

Is LSA and topic clustering easier in European languages similar to English?

I was watching a talk on latent semantic analysis and the speaker described experience applying LSA and REALLY messy data. He concluded that it demonstrated the difficulty of disambiguation of ...
0
votes
1answer
96 views

Time-delayed neural networks - Learning the “max” layer

In Unified Architecture for NLP paper time-delayed neural network proposed as a way to deal with variable length input. Input window slides over sequence and label each output with "time", then next ...
5
votes
1answer
206 views

Using text mining/natural language processing tools for econometrics

I am not sure whether this question is fully appropriate here, if not, please delete. I am a grad student in economics. For a project which investigates issues in social insurances, I have access to ...
2
votes
1answer
104 views

Markov chain getting stuck due to insufficient data samples

There is a lot of theory on Markov models and output generation out there, but I cannot locate any information on models getting stuck. I'm trying to create a model of a data set using a Markov ...
7
votes
1answer
2k views

Intutive difference between hidden Markov models and conditional random fields

I understand that HMM are generative models, and CRF are discriminative models. I also understand how CRFs' are designed and used. What I do not understand is how they are different from HMMs'? I read ...
3
votes
0answers
141 views

Latent Semantic Analysis - Co-occurrence of words

Let $A[n\times m]$ represents the term-document matrix, where, $n$ is the number of terms and $m$ is the number of documents. This matrix can be composed into 3 matrices (SVD decomposition) such as, ...
0
votes
3answers
244 views

Understanding the derivation of an equation in LDA modeling

When reading the derivation of LDA models, I usually get the following equations. I do not quite understand the second step, where $p(\mathbf{z}_{-i},\mathbf{w}|\alpha,\beta)$ was removed. Is that ...
1
vote
1answer
908 views

Loop over Tokens in RapidMiner's Text Processing Plugin

is there any possibility to iterate over the tokens of a text document within RapidMiner? My first try was to window the document after tokenisation. But this seems very complicated. I'm doing this ...
2
votes
1answer
145 views

How to extract ngrams from ambigous text after lemmatization?

After lemmatization of text I have a sequence of sets of lemmas, because every word can correspond to more than one lemma. How should I extract ngram statistics based on that? The only thing that ...