Refers to a subset of data mining concerned with extracting information from data in the form of text by recognizing patterns. The goal of text mining is often to classify a given document into one of a number of categories in an automatic way, and to improve this performance dynamically, making it ...

learn more… | top users | synonyms

1
vote
1answer
23 views

Text mining - Match product title with a description

I have a file with a list of product descriptions. These product description are long strings. Eg "These blue pants are a resistant and comfortable product for tracking and ciclying. In the picture we ...
1
vote
1answer
26 views

Is it necessary to lexicalize the text corpus before applying lda?

While going through a sample lda example code in R R code for topic modelling, the ...
0
votes
0answers
10 views

Difference between Bag of words and Vector space model

I am searching for the intuitive difference between Bag-of-words and vector space model? Is there any relationship exists between bag-of-words and vector space model. I tried searching but couldn't ...
6
votes
1answer
121 views
+50

Automatic keyword extraction: using cosine similarities as features

I've got a document-term matrix $M$, and now I would like to extract keywords for each documents with a supervised learning method (SVM, Naive Bayes, ...). In this model, I already use Tf-idf, Pos ...
0
votes
0answers
20 views

Use Word2Vec to generate a synthetic text dataset

I'm trying to create a realistic set of document like text datasets. Is there any known way to implement the word2vec representation of words in order to manipulate such text?
0
votes
1answer
18 views

Implement a Bigram Latent Dirichlet Allocation (LDA) for Topic Modeling

I'm trying to implement Latent Dirichlet Allocation (LDA) on a bigram language model. This is described in Topic Modeling: Beyond Bag-of-Words by Hanna Wallach et al. I'm trying to easily implement ...
0
votes
0answers
13 views

Fraud detection use text mining [closed]

I have 1 million unstructured text documents about the clients' information with metadata about the client name, viewers, location in the cloud. I want to fraud detection. How can I detected the ...
1
vote
2answers
31 views

How to find similar documents in a big data set

I have many text text documents and my goal is to find similar documents. Apparently it is a clustering type of question and LDA (Latent Dirichlet Allocation) is a good candidate to do that. However ...
0
votes
0answers
38 views

convert 20 category problem into a set of pairwise classifcations

i am struggling to convert a 20 category problem into a set of pairwise classifications. I have separated the categories into 4 classes and calculated the size of the set to be 60. Total = (k-1) x n ...
0
votes
0answers
18 views

Including Feature Selection in Cross Validation - Application to Bag of Words

I am working on a prediction problem where I was given a 6,000 record dataset with the value of the dependent variable included ("train"), and a 2,000 record dataset with the same independent ...
-1
votes
2answers
23 views

Data representation of text clustering system? [on hold]

I'm working on a system that should cluster text based data. However, I'm quite new to NLP domain and can't quite get my head around how this data can be represented numerically. Thus assuming I'm ...
0
votes
0answers
4 views

r : Need content_transformer() called by tm_map() to change non-letters to spaces [migrated]

In the following code, any characters matching "/|@| \|") will be changed to a space. ...
0
votes
0answers
8 views

r : Why is findAssocs() not working? (at all) [migrated]

findAssocs() is not working, as is seen below. "Lucid" and "dreaming" occur together quite often in the book. The corpus is a single document, the text version of a book. Does this function require ...
1
vote
0answers
19 views

Why is removeSparseTerms() not doing anything? [closed]

This has me very, very puzzled... removeSparseTerms() is not removing any terms. ...
0
votes
0answers
25 views
0
votes
0answers
50 views

R:Text Analysis and Classify as type

I am new to R and Analysis, I have content set (emails) that are stored as csv file that is of more than 1000 rows(more than one email content in a row) , these are been imported to R and have been ...
1
vote
0answers
25 views

How Linear SVM works for Text Classification

I am working on text classification problem with Linear SVM. I have some basic knowledge on SVM. I am looking for information on how exactly SVM works for text classification problems, i.e. its ...
2
votes
2answers
54 views

Word2Vec : Interpretation of Subtraction or addition of vectors

I am curious, what does subtracting vectors, as in [man – woman] do in regards to Google's word2vec calculation of analogy ? Is this a measure of how different the two vectors are? So is man – woman ...
2
votes
1answer
475 views

Why is n-gram used in text language identification instead of words?

In two popular language identification libraries, Compact Language Detector 2 for C++ and language detector for java, both of them used (character based) n-grams to extract text features. Why is a ...
0
votes
0answers
11 views

References for learning text stemming

I am trying to learn and experiment with text stemming. My ultimate goal is knowledge extraction from scientific text and corpus with emphasis on contextually multiplicity. But text stemming and ...
0
votes
0answers
11 views

Evaluating annotations with gold standard

I want to evaluate the following: I have a gold set with documents wherefore the meta data is annotated; this means for an article i have annotated what the author is, what the title is, etc.. I now ...
2
votes
1answer
63 views

How to increase the performance of random forest classifier?

I have a text classification task. These are the metrics for different languages at present: class1: 0.6823 class2: 0.7450 class3: 0.66 class4: 0.6719 How can I ...
0
votes
1answer
19 views

Semistructured document classification

I am trying to cluster products based on the text descriptions of the products. I have millions of products. The nature of the products could be hierarchical. i.e; Clothing will have T-Shirts & ...
0
votes
0answers
136 views

Lyrics classification with WEKA

I have a data set of roughly 62.000 lyrics, and I would like to label them as English or not English, because in the next step I would only need the English ones. I have understood that WEKA is a ...
0
votes
1answer
29 views

Which algorithm is best to categorize forum questions? [closed]

I want to categorize users based on the question they post in forums like wordpress, drupal etc. I am stuck at initial step to proceed. Please help me out !!!
0
votes
0answers
22 views

Good baseline algorithm for text-related machine learning project

I'm working on a machine learning project aimed at predicting the quality/helpfulness of a review. For each review in the dataset, I have the review text, a number 'm' for the number of people who ...
0
votes
1answer
24 views

Finding words belonging to a topic

Consider forum posts or any text where we'd be interested in finding out related words, given the data. What would be a solution for creating a topic cluster based on this data? E.g. We are interested ...
2
votes
1answer
37 views

What does it mean for Latent Dirichlet Allocation results to be “good”?

In most paper, Latent Dirichlet Allocation (LDA) model is used for clustering, and the value of $K$ is trained manually (e.g. http://astro.temple.edu/~tua95067/grbovic_cikm.pdf). They claim that this ...
0
votes
1answer
19 views

What are distinctive terms?

Here $n$ is the number of distinctive terms in document $d$. What is the meaning of distinctive? My guess is that it's terms that remain after filtering document from terms that aren't necessary, ...
0
votes
0answers
31 views

Deeplearning for Text Classification

I'm looking for pointers to introductory tutorials on deep learning and text classification. Thanks.
0
votes
1answer
48 views

Feature normalization in Text Classification

I'm doing Text Classification in R, and my initial features are just word frequency inside a Document. For example: ...
0
votes
1answer
23 views

Features Vectors to build classifier to detect subjectivity

I am trying to build a classifier to detect subjectivity. I have text files tagged with subjective and objective . I am little lost with the concept of features creation from this data. I have found ...
1
vote
0answers
24 views

How to understand the patterns of section names in a resume?

recently I am doing some text mining works with resumes. The objective is to divide the resume into several sections based on its headings and contents and then classify it for required jds. Eg. We ...
1
vote
1answer
98 views

Popular named entity resolution software

I am working on a project and need to extract persons' names from a large amount of documents. This task should belong to the named entity resolution problem. What are currently some of the most ...
0
votes
0answers
34 views

SVM for an unbalanced textual dataset?

I have a text classification task, currently I can classify the data with very poor precision. This are the scores: ...
0
votes
0answers
47 views

Handling sparse document term matrix

I am working with a corpus of several thousand documents (41,732) however the documents tend to be short (the median number of terms per document is 3) resulting in a sparse document term matrix. ...
1
vote
0answers
30 views

Naive Bayes text classification on different cardinality classes

I have written a Naive Bayes classifier (with Laplace smoothing) and am using it to classify text into a few simple classes. However, I found that the classes are not of the same vocab size -- to be ...
1
vote
0answers
36 views

Feature space reduction for tag prediction

[x-post] from stackoverflow. I am writing a ML module (python) to predict tags for a stackoverflow question (tag + body). My corpus is of around 5 million questions with title, body and tags for ...
0
votes
0answers
32 views

What algorithm can give me a probability-ranking on text-classifaction problems

I'm trying to classify e-mails using tokens in the text and headers. What I would like to know is which class a new e-mail most likely belongs to. The answer could be more than one class or could ...
0
votes
0answers
12 views

parsing semi-structured textual data poorly matrix formatted

I have found many approaches in the literature that could deal with my problem (which is parsing poorly formatted textual data having a matrix structure but whose content may vary, headers being ...
1
vote
0answers
32 views

Topic models (LDA), word cooccurances in documents?

I have read on papers that Latent Dirichlet Allocation (LDA) works by identifying word cooccurances in documents. What is confusing me is since LDA uses bag-of-words approach for document ...
1
vote
2answers
398 views

Python vs R for Text Mining Preprocessing

I've been reading some articles on cleaning text data before doing text mining analysis on it. I have experience in both Python and R and am wondering if one of these languages is an obviously better ...
0
votes
0answers
25 views

Normalization of Naive Bayes output

In Scikit-learn documentation it is possible to see that the MultinomialNB estimator has a method called predict-proba in which it has the following description: "Returns the probability of the ...
2
votes
3answers
58 views

A good intro to computational linguistics?

I have a pretty good background in data analysis and statistics in the social sciences, including both frequentist and Bayesian paradigms, and I have recently been introduced to computational ...
3
votes
2answers
137 views

What does “Virgin Data” mean?

I am using RTextTools, which has a function to create container with following syntax: create_container(matrix, labels, trainSize=NULL, testSize=NULL, virgin) ...
0
votes
1answer
130 views

Generating text data for training for doing named entity recognition and extraction

I'm trying to build an algorithm for doing named entity extraction. It goes like this. There is a large set of text documents [communications], from which specific information has to be extracted. The ...
1
vote
1answer
115 views

Text Classification using TfIdf and Bernoulli NB

So, as I am reading about Bernoulli distribution and text classification, I want to understand how Bernoulli uses TfIdf features? Since TfIdf values are within [0-1) but Multivariate Bernoulli assumes ...
1
vote
1answer
64 views

How to find the perplexity of a corpus

The formula of the perplexity measure is: $$p: \left(\frac{1}{\sqrt[n]{p(w_1^n)}}\right)$$ where: $p(w_1^n)$ is: $\prod_{i=1}^n p(w_i)$. If I understand it correctly, this means that I could ...
0
votes
1answer
75 views

A single document as input to LDA?

We use topic modelling usually on a collection of documents - which makes the input. But what if I only have a single document where I want to see the underlying topics in it? I have heard that you ...
0
votes
1answer
26 views

Multiple labels in supervised learning algorithm

I have a corpus of text with a corresponding topics. For example "A rapper Tupac was shot in LA" and it was labelled as ...