Statistical analysis of texts expressed in languages spoken or written by people, such as English or Norwegian.

learn more… | top users | synonyms

1
vote
1answer
13 views

How can I improve feature selection for my Naive Bayes Classifier?

I am classifying companies into two classes ( a particular business type, or not that business type ), using a Naive Bayes Classifier. Specifically, I'm using PHP and PHP NLP Tools. I have two ...
1
vote
0answers
14 views

Natural Language measure of obfuscation

I have some experience with sentiment analysis in natural language processing, but want to learn some new algorithms and techniques for a project I am working on. In particular, I am interested in a ...
1
vote
0answers
16 views

Different Methods for clustering skills in text

Consider a talent pool in which each member has some set of skills. Some of these talent are submitted to orders as potential candidates of which one is selected. It is reasonable to assume that the ...
4
votes
3answers
118 views

Is it feasible to use k-Nearest Neighbours to identify text language?

I have seen various language identification libraries that claim to use naive Bayes classifier for text language identification, like CLD2 and language detector, but not any library that uses other ...
0
votes
0answers
4 views

What are good tutorial on Weighted Finite Automata?

I would especially appreciate papers, books or tutorials with source code already available. Currently I'm reading "Spectral Learning Techniques for Weighted Automata, Transducers, and Grammars" by ...
0
votes
0answers
30 views
1
vote
0answers
23 views

What machine learning algorithm should I choose to fill in blanks from context?

I have a project where I need to be able to fill in a missing word given a few words of context. In other words, suppose I have a sentence: I went ____ the store. I want to be able to deduce ...
2
votes
1answer
515 views

Why is n-gram used in text language identification instead of words?

In two popular language identification libraries, Compact Language Detector 2 for C++ and language detector for java, both of them used (character based) n-grams to extract text features. Why is a ...
0
votes
0answers
11 views

References for learning text stemming

I am trying to learn and experiment with text stemming. My ultimate goal is knowledge extraction from scientific text and corpus with emphasis on contextually multiplicity. But text stemming and ...
1
vote
1answer
43 views

How to get corpus used for word-frequencies for languages other than english?

I have taken a corpus of English for finding the word frequency in giving recommendation for spelling mistakes. I just used simple : Edit Distance between ...
6
votes
1answer
44 views

Has there been a project to apply machine learning to generation of indices for books?

Generating an index for a textbook is a tedious task. Can one automate it with machine learning? Are there any references to previous attempts in the literature to do this?
0
votes
0answers
25 views

What statistical/statistical analysis can I use to extract information from set of words?

I have long (thousands) lists of words set. A set of words is usually combination of a number and unit of measurement, but it can also be a combination of words. What kind of analysis can I use to ...
4
votes
1answer
55 views

How to make an effective sampling from a database of text documents?

Problem: I want to know methods to perform an effective sampling from a database. The size of the database is about 250K text documents and in this case each text ...
0
votes
1answer
45 views

Question about Continuous Bag of Words

I'm having trouble understanding this sentence: ...
2
votes
1answer
48 views

How to handle changing input vector length with neural networks

I want to train a neural network with a sequence of character as an input vector. Learning examples have different length and for this reason I don't know how to represent them. Let's say I have two ...
0
votes
0answers
76 views

Sentiment Analysis - How should I handle negatively biased word list length?

I'm implementing a simple sentiment analysis algorithm where the authors of the paper have a word list for positive and negative words and simply count the number of occurrences of each in the ...
1
vote
0answers
29 views

How to understand the patterns of section names in a resume?

recently I am doing some text mining works with resumes. The objective is to divide the resume into several sections based on its headings and contents and then classify it for required jds. Eg. We ...
0
votes
1answer
37 views

Word-frequency and statistics

I am new to statistics and am wondering how I can apply it in linguistics. There is a conjunction in a corpus that 902 times (.91) conjoins sub-clausal units, and 91 (.09) times clausal units. How ...
0
votes
0answers
18 views

Using LIWC for ANOVA and pairwise analyses

I want to use a tool called Linguistic Inquiry and Word Count (LIWC) to analyze text data. As far as I know, it counts the number of words found in a given text that match a set of dimensions (e.g., ...
2
votes
1answer
29 views

Comparing corpus complexities

I would like to compare how complex (varied or predictable) are my three corpora. They are from different topics, so some vocabulary is different, some is the same. Looking at one of the data sets ...
0
votes
0answers
13 views

How alignment between words are generated by IBM model1?

I implemented the translation model IBM1. As a result, I got the translation table P(targetWord|sourceWord) wich is ok. I want also to obtain the alignment of words in the corpus that I used in order ...
2
votes
3answers
59 views

A good intro to computational linguistics?

I have a pretty good background in data analysis and statistics in the social sciences, including both frequentist and Bayesian paradigms, and I have recently been introduced to computational ...
1
vote
0answers
18 views

How to learn (arbitrary) constraints for selecting the best candidate from a group?

In my classification problem, each instance is a group of possibly hundreds of candidates, from which only one should receive the label $True$ and the remainder the label $False$. For example, in ...
0
votes
1answer
131 views

How to standardize text data for training Neural Networks?

I want to train neural network with text data(natural language) as input for classification purpose. One way for standardizing text data for neural network is to use N-GRAM/SKIP-GRAM representation ...
1
vote
1answer
73 views

How to find the perplexity of a corpus

The formula of the perplexity measure is: $$p: \left(\frac{1}{\sqrt[n]{p(w_1^n)}}\right)$$ where: $p(w_1^n)$ is: $\prod_{i=1}^n p(w_i)$. If I understand it correctly, this means that I could ...
2
votes
1answer
25 views

determining significance of term use

Thing one: feel free to RTFM me: I'm definitely looking for search-able terms or background reading. Our situation is this: we have a set of 140 reviewers and 20 elements. Each reviewer reviews each ...
1
vote
1answer
48 views

Neural language model training - stochastic vs batch

Dealing with a very basic neural language model: 3 words of context, vector size 100, one hidden layer size 200, vocabulary size 1000, predicting the next word with a softmax output layer. Previously ...
2
votes
0answers
17 views

How to measure how 'well' I am matching Google keywords?

For google keywords you can bid on a broad match. For example let's say I bid on the keyword 'best hamburger' and somebody searches 'What sort of beef makes the best hamburger?' and 'eat best ...
0
votes
0answers
34 views

Hashing functions in NLP

I have been reading a lot of papers about nlp which use the hashing trick, and I came across a lot of sentences like : "We take k hashing functions to hash words or bi-grams". And after that they ...
0
votes
1answer
59 views

How do search engines generate related searches?

I would like to know how search engines like Bing generate related searches when the user starts typing into the search box. From what I gather, there has to be some sort of a ranking algorithm where ...
0
votes
1answer
37 views

How can I show that my sample is random and a good representation of the population?

Okay, I'm looking at a population of user reviews. I have collected a random sample of the reviews and studied the trends of the words used and sentence structure. How do I make inferences about ...
0
votes
0answers
62 views

Trying to understand the basics of a mixed-effects logistic regression model for a 10-step continuum

I am trying understand how to correctly build a mixed-effects logistic regression model in R. I believe my model is pretty simple and straight forward but I'm lacking in experience and uncertain I'm ...
1
vote
0answers
150 views

Any advice on how to improve my accuracy rate in text classification?

I'm trying to do a text classification task. Here are some specs: Context file size = 1M+ documents already labeled Number of top-labels = 17 Number of sub-labels = around 130 Each document is ...
0
votes
0answers
28 views

Unsupervised feature learning from raw text as a previous step for clasification?

I have a corpus of 2500 opinions, is it posible to use scikitĀ“s restricted boltzmann machine implementation to extract a feature vector as a previous step to a classification task?. What aproach do i ...
2
votes
2answers
84 views

Weighting words based on position in text

I'm currently working on semantic analysis and had a question about text organization and structure. Are there any algorithms, or statistical / machine-learning models that weight the importance of a ...
1
vote
1answer
142 views

How to handle unseen features in a Naive Bayes classifier?

I am writing a naive bayes classifier for a text classification problem. I have a bunch of words and an associated label: ...
0
votes
1answer
38 views

Feature learning with a deep learning aproach?

How to create a feature vector from text with a deep learning aproach?. Im new at this topic, could anybody advice me where to start and how to aproach this task?.
4
votes
0answers
154 views

Understanding Singular Value Decomposition in the context of LSI

My question is generally on Singular Value Decomposition (SVD), and particularly on Latent Semantic Indexing (LSI). Say, I have $ A_{word \times document} $ that contains frequencies of 5 words for ...
0
votes
0answers
178 views

What algorithms should I use to perform job classification based on resume data?

Note that I am doing everything in R. The problem goes as follow: Basically, I have a list of resumes (CVs). Some candidates will have work experience before and some don't. The goal here is to: ...
2
votes
1answer
102 views

Calculating Perplexity

In the Coursera NLP course , Dan Jurafsky calculates the following perplexity: Operator(1 in 4) Sales(1 in 4) Technical Support(1 in 4) 30,000 names(1 in 120,000 each) He says the Perplexity is 53. ...
2
votes
1answer
46 views

Question about the probability chain rule

I've understood from this: Is this a correct statement of the probability chain rule? that in the chain rule for probability, conditioning can be done on different variables. I was wondering what ...
2
votes
1answer
624 views

Naive Bayes with unbalanced classes

As a part of a project for the university is should train a Naive Bayes classifier to classify question and answers in three different categories, the task should be easy since that the 3 classes are ...
2
votes
1answer
196 views

Visualizing Mutual Information Against TF-IDF for Text Corpus Data

I'm working on a data visualization project for the semester and have decided to work with a corpus of discussion forum data focused around debate over political issues (available here). I'm ...
1
vote
1answer
139 views

Hidden Markov Model: Predict observation sequence from state sequence

Given a transition matrix, starting probability, means and covariances Is it possible to predict the most likely observed sequence for a given state sequence using the above details? If yes, how? ...
1
vote
0answers
69 views

How to extract structured information from a text string?

I have a text string containing unstructured data and I would like to analyze it in order to extract structured information. In particular, this text string specifies when a service is operational ...
0
votes
1answer
103 views

Multiple Bernoulli and Multinomial Distirbution

It's well known that language can be modeled by Multinomial distribution and Multiple Bernoulli distribution. So far I don't see any advantage of Multiple Bernoulli distribution representation over ...
5
votes
0answers
82 views

Expectation Maximization Clarification

I found very helpful tutorial regarding EM algorithm. The example and the picture from the tutorial is simply brilliant. Related question about calculating probabilities how does expectation ...
0
votes
1answer
59 views

Improvement on duplicating instances

I have a task of Relationship extraction. There are some set of predefined relations in the corpus. I need to train classifier to recognize the type of relation or the lack of relation between every ...
1
vote
2answers
242 views

Most important journals in data mining/ML, NLP and IR?

Can you please provide with me with the names of the most important journals in data mining, machine learning, natural language processing and information retrieval?
1
vote
0answers
93 views

Calculating and Normalizing ngram relevancy scores from free text extraction

I currently look for a set of ngrams in many sets of documents to establish a relevancy score for each set - eg. I look for the n-gram "adhesive tape" in ~1M sets of 1-500 documents. The values I ...