Refers to a subset of data mining concerned with extracting information from data in the form of text by recognizing patterns. The goal of text mining is often to classify a given document into one of a number of categories in an automatic way, and to improve this performance dynamically, making it ...

learn more… | top users | synonyms

0
votes
0answers
16 views

Extracting addresses from HTML invoices [on hold]

I am given a set of a couple of thousands of invoices in HTML format. They are invoices from many years and different products, so they really differ in layout. I need to extract the address ...
1
vote
0answers
22 views

Topic models (LDA), word cooccurances in documents?

I have read on papers that Latent Dirichlet Allocation (LDA) works by identifying word cooccurances in documents. What is confusing me is since LDA uses bag-of-words approach for document ...
1
vote
2answers
76 views

Python vs R for Text Mining Preprocessing

I've been reading some articles on cleaning text data before doing text mining analysis on it. I have experience in both Python and R and am wondering if one of these languages is an obviously better ...
1
vote
0answers
29 views

What's the difference between TF-IDF and DF-ICF? [closed]

I have read that TF-IDF deals with terms and DF-ICF deals with documents. But I need an example to understand it well. Consider the use of these in twitter: how would they be applied with twitter? ...
0
votes
0answers
18 views

Normalization of Naive Bayes output

In Scikit-learn documentation it is possible to see that the MultinomialNB estimator has a method called predict-proba in which it has the following description: "Returns the probability of the ...
2
votes
3answers
48 views

A good intro to computational linguistics?

I have a pretty good background in data analysis and statistics in the social sciences, including both frequentist and Bayesian paradigms, and I have recently been introduced to computational ...
3
votes
2answers
90 views

What does “Virgin Data” mean?

I am using RTextTools, which has a function to create container with following syntax: create_container(matrix, labels, trainSize=NULL, testSize=NULL, virgin) ...
0
votes
1answer
97 views

Generating text data for training for doing named entity recognition and extraction

I'm trying to build an algorithm for doing named entity extraction. It goes like this. There is a large set of text documents [communications], from which specific information has to be extracted. The ...
1
vote
1answer
45 views

Text Classification using TfIdf and Bernoulli NB

So, as I am reading about Bernoulli distribution and text classification, I want to understand how Bernoulli uses TfIdf features? Since TfIdf values are within [0-1) but Multivariate Bernoulli assumes ...
0
votes
0answers
19 views

How to find the perplexity of a corpus

The formula of the perplexity measure is thus: $ p: \left(\frac{1}{\sqrt[n]{p(w_1^n)}}\right) $ where: $p(w_1^n)$ is: $\prod_{i=1}^n p(w_i)$ If I understand it correctly, this means that I ...
0
votes
1answer
41 views

A single document as input to LDA?

We use topic modelling usually on a collection of documents - which makes the input. But what if I only have a single document where I want to see the underlying topics in it? I have heard that you ...
0
votes
1answer
24 views

Multiple labels in supervised learning algorithm

I have a corpus of text with a corresponding topics. For example "A rapper Tupac was shot in LA" and it was labelled as ...
2
votes
1answer
24 views

determining significance of term use

Thing one: feel free to RTFM me: I'm definitely looking for search-able terms or background reading. Our situation is this: we have a set of 140 reviewers and 20 elements. Each reviewer reviews each ...
0
votes
0answers
21 views

The best algorithm for short documents clustering

I have a corpus of short text documents. Each document is an automatic recognized phone conversation (a dialog) from a large call center. The texts are not clean and have lots of grammar and other ...
5
votes
3answers
271 views

How would you categorize / extract information out of job descriptions?

I have a bunch of job descriptions entered by users. There are all sort of misspells and bad data. i.e: ...
1
vote
1answer
63 views

How to prepare a dataset for text classification

I would like to compare some algorithms for performing sentiment classification (Naive Bayes, SVM, and ...
0
votes
0answers
38 views

VW multiclass classification

I am new to vw and trying to do a multiclass text classification with 18 classes. features are unigram, bigram and trigrams. Total features are around 1.4 million Total training examples 35 million ...
2
votes
0answers
16 views

How to measure how 'well' I am matching Google keywords?

For google keywords you can bid on a broad match. For example let's say I bid on the keyword 'best hamburger' and somebody searches 'What sort of beef makes the best hamburger?' and 'eat best ...
0
votes
1answer
55 views

In natural language processing (NLP), how do you make an efficient dimension reduction?

In NLP, it's always the case that the dimension of the features are very huge. For example, for one project at hand, the dimension of features is almost 20 thousands (p = 20,000), and each feature is ...
1
vote
0answers
13 views

Calculate association between text documents [duplicate]

I've got 6000 reports. For each report, I've got a vector with the keywords in it and a vector with the tokens of its abstract. Now I want to calculate some association between those two sets. Is ...
0
votes
1answer
40 views

Vector Space Model for Online News Clustering

I am trying to automatically cluster news articles based on their content. I need this algorithm to be online and simply group news articles related to the same story as they arrive. The common ...
2
votes
0answers
70 views

Has the reported state-of-the-art performance of using paragraph vectors for sentiment analysis been replicated?

I was impressed by the results in the ICML 2014 paper "Distributed Representations of Sentences and Documents" by Le and Mikolov. The technique they describe, called "paragraph vectors", learns ...
0
votes
1answer
18 views

Easy way of indicating the likelihood of a text?

I am trying to find out a simple way of quantifying "how much sense a text paragraph makes". The measure is not necessarily fail-proof, but it has to be somewhat simple. One example measure that ...
3
votes
1answer
76 views

Methodology for standardizing names

We have a great amount of data with user generated names for company names (think of bills where the company wrote their own name). In our data cleaning phase, we clustered the different companies ...
0
votes
0answers
25 views

Dataset with mixed structured/unstructured data

I am looking for some mixed-type dataset with free-form textual features along with some structural features (preferably continuous) - labeled, for supervised setting (classification or prediction). ...
1
vote
2answers
92 views

How to transform test set to the PCA space of the training set, if the features in train and test are different?

I'm working on a text classification project, and I want to reduce the tf-idf matrix dimension with Principal Component Analysis (PCA) and then train my model with this, which is pretty ...
3
votes
0answers
31 views

TF-IDF Matrix and Regression

I am trying to build a regression model based on some tweets that my company put on our company feed. I would like to transform all of the tweets, and use them to tell me which word(s) were most ...
1
vote
0answers
18 views

Is Social Network Analysis or NER the best way to create a semantic graph?

I am planning to create a semantic graph by creating an automatic ontology. I want to know which is the best process to do it. Doing social network analysis to create people, relationships, likes, ...
1
vote
0answers
107 views

Any advice on how to improve my accuracy rate in text classification?

I'm trying to do a text classification task. Here are some specs: Context file size = 1M+ documents already labeled Number of top-labels = 17 Number of sub-labels = around 130 Each document is ...
0
votes
0answers
22 views

Latent Semantic Analysis: scale representation of documents?

After performing SVD on the term-document matrix, the right eigenvectors correspond to the representation of documents in the reduced concept space. In order to use this for say text classification, ...
0
votes
1answer
281 views

How to use k-fold cross validation in naive bayes classifier?

I'm trying to classify text using naive bayes classifier, and also want to use k-fold cross validation to validate the result of classification. But I'm still confused how to use the k-fold cross ...
0
votes
0answers
42 views

R Clustering Evaluation (Adaptive Kmeans)

i know there are several threads about this topic, but most i read, most i get confused. I'm doing a project that consists in clustering some data (news articles). I used adaptive Kmeans ...
3
votes
1answer
135 views

can we generate a random words from English letters that follow the bigram of the English language

The main issue is that several research building their solution of detecting and classifying English language is based on bigram distribution. However, I would like to know if it possible to generate ...
1
vote
0answers
77 views

In Kneser-Ney smoothing, how are unseen words handled?

From what I have seen, the (second-order) Kneser-Ney smoothing formula is in some way or another given as $ \begin{align} P^2_{KN}(w_n|w_{n-1}) &= \frac{\max \left\{ C\left(w_{n-1}, w_n\right) - ...
1
vote
0answers
131 views

Naive Bayes and text classification: which probability model and vectorizer combination makes sense?

I am wondering which combinations of Naive models can be paired with different vectorizing methods so that it makes sense. Let's say we have a simple binary spam-classification task. Multinomial ...
0
votes
0answers
26 views

choosing best value for N when using N-Gram approach

the question is quite general, but I am doing a research related to supervised machine learning to classify two set of characters into two categories. in fact, I want to compute some measures of ...
1
vote
1answer
155 views

Using topic words generated by LDA to represent a document

I want to do document classification by representing each document as a set of features. I know that there are many ways: BOW, TFIDF, ... I want to use Latent Dirichlet Allocation (LDA) to extract ...
0
votes
0answers
52 views

Unsupervised machine learning with numerical and text data

My dataset has numerical variables and one "Note" variable with around a paragraph of text. I'd like to classify the data with an unsupervised learning algorithm. I've done some searching and one ...
0
votes
0answers
21 views

Asking tweet classification

I want to ask you the process to classify the tweet data. Now, I am working to Twitter data but i have confuse how to classify the tweet data using Mallet Tool. Example; I have 200,000 tweets. The ...
1
vote
0answers
25 views

Extracting city name from free text?

I'm having a set of free text from web. Since the users type their location in that field, we have many un-normalized city names. For example, "Shanghai, China" "China, ShangHai" might mean the ...
0
votes
0answers
18 views

How do I extract a keyword plus its corresponding number out of a text

I am finding my way around in the different text mining tools, but I can't find the technique that does the following: Extract out of the follwing advertisement texts ...
0
votes
1answer
90 views

Using a Dictionary for Text Mining

I want to use a dictionary to analyze a text with text mining mechanisms. So I'm looking for a dictionary with scores for each words, so for example the word "cool", +1, gets a positive score, the ...
0
votes
0answers
13 views

Mallet API installation

I am trying to use the mallet API to classify text. Previously created classifier using the command line tool. I don't know to use Mallet packages using Java. Thank you
0
votes
0answers
5 views

Get positions/values from heat costs bill

for a project I need to extract values from customers yearly heat costs bill. The customer takes a photo of the bill and the program should extract the values heating period of the billing, type of ...
1
vote
0answers
36 views

CoreNLP SemanticGraph - search for edges with specific lemmas

I'm using Stanford CoreNLP's dependency parser, and wondering how to make a generic search for SemanticEdge(s) with specific head lemma, dependent lemma, and lexical relationship. For example, if I ...
0
votes
0answers
8 views

automatic assign class name based on text

My question is , I have a set of plain text , i want to create category based on the text. Eg: i have written something about Soup recepie then the algorithm must create a category called Food. After ...
0
votes
1answer
132 views

Multi-class Confusion Matrix to Binary confusion matrix

i know the main concepts of data/text mining but i used them mainly in binary classification problems (just two classes). i am now dealing with a problem with 8 classes and i am atruggling how to ...
0
votes
0answers
79 views

Using relative frequency for Euclidean and cosine distance (dissimilarity)

How to calculate the Euclidean distance (dissimilarity) between two documents, e.g., D1 and D2 using relative frequency? Here is an example of both cosine and Euclidean distance between two ...
2
votes
0answers
28 views

Finding related words

I have several files, each of which contains unique terms which are related to each other(without sentence structure). So for finding the word relationships I created a dictionary of bi-grams for ...
2
votes
2answers
80 views

Weighting words based on position in text

I'm currently working on semantic analysis and had a question about text organization and structure. Are there any algorithms, or statistical / machine-learning models that weight the importance of a ...