Refers to a subset of data mining concerned with extracting information from data in the form of text by recognizing patterns. The goal of text mining is often to classify a given document into one of a number of categories in an automatic way, and to improve this performance dynamically, making it ...

learn more… | top users | synonyms

-2
votes
1answer
20 views

How to do text clustering for a set of around 10000 messages?

I have around 10000 messages in a variable, i want to form clusters of them based on similarity, so that I can assign some class say 1-10, if 10 clusters are formed and run analysis on them. How can ...
6
votes
1answer
79 views

Bag-of-Words for Text Classification: Why not just use word frequencies instead of TFIDF?

A common approach to text classification is to train a classifier off of a 'bag-of-words'. The user takes the text to be classified and counts the frequencies of the words in each object, followed by ...
1
vote
1answer
50 views

Text analysis : What after term-document matrix?

I am trying to build predictive models from text data. I built document-term matrix from the text data (unigram and bigram) and built different types of models on that (like svm, random forest, ...
0
votes
0answers
18 views

machine learning for a ontology classification problem

I am working on a ontology based classification problem.The main objective was: computing ontology has keywords related to different categories.Each category talks about the domain it is related.For ...
0
votes
0answers
24 views

Probability that a large corpus of text is generated with the same parameters as a subset

Let's say I have a process which generates different words at a set (unknown) frequency per word. I sample this process X times, generating the word "yo" Y times. I then look at a subset of my ...
0
votes
0answers
22 views

How to evaluate and compare two clustering algorithms in R for text mining

I am doing research in R language for text mining. I would like to know how to evaluate and compare two clustering algorithms in R for text mining?
2
votes
2answers
86 views

Supervised keyword extraction: results

I've considered the keyword extraction method as a classification problem (1 = author generated keyword, 0 = no keyword) and I've tried to automatically extract keywords from text document. I've done ...
0
votes
0answers
20 views

How to simulate a multivariate Logistic-Normal distribution in Python

I'm trying to generate a text document using reverse "Correlated Topic Models", which is an advanced version of LDA (Latent Dirichlet Allocation). In this version the topics are generated over a ...
0
votes
0answers
21 views

How do I weight inputs to a regression model so that one figures into the model more than the other?

I have obtained a series of weights from a text mining algorithm. Unfortunately, my algorithm is not capable of doing certain tasks that are too similar without some sort of regression analysis, say ...
1
vote
1answer
39 views

Text mining - Match product title with a description

I have a file with a list of product descriptions. These product description are long strings. Eg "These blue pants are a resistant and comfortable product for tracking and ciclying. In the picture we ...
1
vote
1answer
37 views

Is it necessary to lexicalize the text corpus before applying lda?

While going through a sample lda example code in R R code for topic modelling, the ...
1
vote
0answers
14 views

Difference between Bag of words and Vector space model

I am searching for the intuitive difference between Bag-of-words and vector space model? Is there any relationship exists between bag-of-words and vector space model. I tried searching but couldn't ...
7
votes
1answer
217 views

Automatic keyword extraction: using cosine similarities as features

I've got a document-term matrix $M$, and now I would like to extract keywords for each documents with a supervised learning method (SVM, Naive Bayes, ...). In this model, I already use Tf-idf, Pos ...
0
votes
0answers
32 views

Use Word2Vec to generate a synthetic text dataset

I'm trying to create a realistic set of document like text datasets. Is there any known way to implement the word2vec representation of words in order to manipulate such text?
0
votes
1answer
22 views

Implement a Bigram Latent Dirichlet Allocation (LDA) for Topic Modeling

I'm trying to implement Latent Dirichlet Allocation (LDA) on a bigram language model. This is described in Topic Modeling: Beyond Bag-of-Words by Hanna Wallach et al. I'm trying to easily implement ...
1
vote
2answers
39 views

How to find similar documents in a big data set

I have many text text documents and my goal is to find similar documents. Apparently it is a clustering type of question and LDA (Latent Dirichlet Allocation) is a good candidate to do that. However ...
0
votes
0answers
38 views

convert 20 category problem into a set of pairwise classifcations

i am struggling to convert a 20 category problem into a set of pairwise classifications. I have separated the categories into 4 classes and calculated the size of the set to be 60. Total = (k-1) x n ...
0
votes
0answers
22 views

Including Feature Selection in Cross Validation - Application to Bag of Words

I am working on a prediction problem where I was given a 6,000 record dataset with the value of the dependent variable included ("train"), and a 2,000 record dataset with the same independent ...
0
votes
0answers
4 views

r : Need content_transformer() called by tm_map() to change non-letters to spaces [migrated]

In the following code, any characters matching "/|@| \|") will be changed to a space. ...
1
vote
0answers
33 views

Why is removeSparseTerms() not doing anything? [closed]

This has me very, very puzzled... removeSparseTerms() is not removing any terms. ...
0
votes
0answers
26 views
0
votes
0answers
51 views

R:Text Analysis and Classify as type

I am new to R and Analysis, I have content set (emails) that are stored as csv file that is of more than 1000 rows(more than one email content in a row) , these are been imported to R and have been ...
1
vote
1answer
43 views

How Linear SVM works for Text Classification

I am working on text classification problem with Linear SVM. I have some basic knowledge on SVM. I am looking for information on how exactly SVM works for text classification problems, i.e. its ...
2
votes
2answers
69 views

Word2Vec : Interpretation of Subtraction or addition of vectors

I am curious, what does subtracting vectors, as in [man – woman] do in regards to Google's word2vec calculation of analogy ? Is this a measure of how different the two vectors are? So is man – woman ...
2
votes
1answer
510 views

Why is n-gram used in text language identification instead of words?

In two popular language identification libraries, Compact Language Detector 2 for C++ and language detector for java, both of them used (character based) n-grams to extract text features. Why is a ...
0
votes
0answers
11 views

References for learning text stemming

I am trying to learn and experiment with text stemming. My ultimate goal is knowledge extraction from scientific text and corpus with emphasis on contextually multiplicity. But text stemming and ...
0
votes
0answers
12 views

Evaluating annotations with gold standard

I want to evaluate the following: I have a gold set with documents wherefore the meta data is annotated; this means for an article i have annotated what the author is, what the title is, etc.. I now ...
2
votes
1answer
86 views

How to increase the performance of random forest classifier?

I have a text classification task. These are the metrics for different languages at present: class1: 0.6823 class2: 0.7450 class3: 0.66 class4: 0.6719 How can I ...
0
votes
1answer
19 views

Semistructured document classification

I am trying to cluster products based on the text descriptions of the products. I have millions of products. The nature of the products could be hierarchical. i.e; Clothing will have T-Shirts & ...
0
votes
0answers
139 views

Lyrics classification with WEKA

I have a data set of roughly 62.000 lyrics, and I would like to label them as English or not English, because in the next step I would only need the English ones. I have understood that WEKA is a ...
0
votes
1answer
31 views

Which algorithm is best to categorize forum questions? [closed]

I want to categorize users based on the question they post in forums like wordpress, drupal etc. I am stuck at initial step to proceed. Please help me out !!!
0
votes
0answers
26 views

Good baseline algorithm for text-related machine learning project

I'm working on a machine learning project aimed at predicting the quality/helpfulness of a review. For each review in the dataset, I have the review text, a number 'm' for the number of people who ...
0
votes
1answer
27 views

Finding words belonging to a topic

Consider forum posts or any text where we'd be interested in finding out related words, given the data. What would be a solution for creating a topic cluster based on this data? E.g. We are interested ...
2
votes
1answer
38 views

What does it mean for Latent Dirichlet Allocation results to be “good”?

In most paper, Latent Dirichlet Allocation (LDA) model is used for clustering, and the value of $K$ is trained manually (e.g. http://astro.temple.edu/~tua95067/grbovic_cikm.pdf). They claim that this ...
0
votes
1answer
21 views

What are distinctive terms?

Here $n$ is the number of distinctive terms in document $d$. What is the meaning of distinctive? My guess is that it's terms that remain after filtering document from terms that aren't necessary, ...
0
votes
0answers
36 views

Deeplearning for Text Classification

I'm looking for pointers to introductory tutorials on deep learning and text classification. Thanks.
0
votes
1answer
62 views

Feature normalization in Text Classification

I'm doing Text Classification in R, and my initial features are just word frequency inside a Document. For example: ...
0
votes
1answer
28 views

Features Vectors to build classifier to detect subjectivity

I am trying to build a classifier to detect subjectivity. I have text files tagged with subjective and objective . I am little lost with the concept of features creation from this data. I have found ...
1
vote
0answers
27 views

How to understand the patterns of section names in a resume?

recently I am doing some text mining works with resumes. The objective is to divide the resume into several sections based on its headings and contents and then classify it for required jds. Eg. We ...
1
vote
1answer
124 views

Popular named entity resolution software

I am working on a project and need to extract persons' names from a large amount of documents. This task should belong to the named entity resolution problem. What are currently some of the most ...
0
votes
0answers
37 views

SVM for an unbalanced textual dataset?

I have a text classification task, currently I can classify the data with very poor precision. This are the scores: ...
0
votes
0answers
54 views

Handling sparse document term matrix

I am working with a corpus of several thousand documents (41,732) however the documents tend to be short (the median number of terms per document is 3) resulting in a sparse document term matrix. ...
1
vote
0answers
31 views

Naive Bayes text classification on different cardinality classes

I have written a Naive Bayes classifier (with Laplace smoothing) and am using it to classify text into a few simple classes. However, I found that the classes are not of the same vocab size -- to be ...
1
vote
0answers
41 views

Feature space reduction for tag prediction

[x-post] from stackoverflow. I am writing a ML module (python) to predict tags for a stackoverflow question (tag + body). My corpus is of around 5 million questions with title, body and tags for ...
0
votes
0answers
33 views

What algorithm can give me a probability-ranking on text-classifaction problems

I'm trying to classify e-mails using tokens in the text and headers. What I would like to know is which class a new e-mail most likely belongs to. The answer could be more than one class or could ...
0
votes
0answers
12 views

parsing semi-structured textual data poorly matrix formatted

I have found many approaches in the literature that could deal with my problem (which is parsing poorly formatted textual data having a matrix structure but whose content may vary, headers being ...
1
vote
0answers
33 views

Topic models (LDA), word cooccurances in documents?

I have read on papers that Latent Dirichlet Allocation (LDA) works by identifying word cooccurances in documents. What is confusing me is since LDA uses bag-of-words approach for document ...
1
vote
2answers
513 views

Python vs R for Text Mining Preprocessing

I've been reading some articles on cleaning text data before doing text mining analysis on it. I have experience in both Python and R and am wondering if one of these languages is an obviously better ...
0
votes
0answers
27 views

Normalization of Naive Bayes output

In Scikit-learn documentation it is possible to see that the MultinomialNB estimator has a method called predict-proba in which it has the following description: "Returns the probability of the ...
2
votes
3answers
59 views

A good intro to computational linguistics?

I have a pretty good background in data analysis and statistics in the social sciences, including both frequentist and Bayesian paradigms, and I have recently been introduced to computational ...