Refers to a subset of data mining concerned with extracting information from data in the form of text by recognizing patterns. The goal of text mining is often to classify a given document into one of a number of categories in an automatic way, and to improve this performance dynamically, making it ...

learn more… | top users | synonyms

-1
votes
0answers
12 views

Python Scikit-learn CountVectorizer throwing ValueError: empty vocabulary [on hold]

I'm trying to extract features from a text document. Here is my code: import sklearn from sklearn.datasets import load_files ...
0
votes
1answer
18 views

Finding words belonging to a topic

Consider forum posts or any text where we'd be interested in finding out related words, given the data. What would be a solution for creating a topic cluster based on this data? E.g. We are interested ...
2
votes
1answer
32 views

What does it mean for Latent Dirichlet Allocation results to be “good”?

In most paper, Latent Dirichlet Allocation (LDA) model is used for clustering, and the value of $K$ is trained manually (e.g. http://astro.temple.edu/~tua95067/grbovic_cikm.pdf). They claim that this ...
0
votes
1answer
18 views

What are distinctive terms?

Here $n$ is the number of distinctive terms in document $d$. What is the meaning of distinctive? My guess is that it's terms that remain after filtering document from terms that aren't necessary, ...
0
votes
0answers
15 views

Deeplearning for Text Classification

I'm looking for pointers to introductory tutorials on deep learning and text classification. Thanks.
0
votes
1answer
22 views

Feature normalization in Text Classification

I'm doing Text Classification in R, and my initial features are just word frequency inside a Document. For example: ...
0
votes
1answer
14 views

Features Vectors to build classifier to detect subjectivity

I am trying to build a classifier to detect subjectivity. I have text files tagged with subjective and objective . I am little lost with the concept of features creation from this data. I have found ...
1
vote
0answers
16 views

How to understand the patterns of section names in a resume?

recently I am doing some text mining works with resumes. The objective is to divide the resume into several sections based on its headings and contents and then classify it for required jds. Eg. We ...
0
votes
1answer
38 views

Popular named entity resolution software

I am working on a project and need to extract persons' names from a large amount of documents. This task should belong to the named entity resolution problem. What are currently some of the most ...
0
votes
0answers
21 views

SVM for an unbalanced textual dataset?

I have a text classification task, currently I can classify the data with very poor precision. This are the scores: ...
0
votes
0answers
19 views

Handling sparse document term matrix

I am working with a corpus of several thousand documents (41,732) however the documents tend to be short (the median number of terms per document is 3) resulting in a sparse document term matrix. ...
1
vote
0answers
24 views

Naive Bayes text classification on different cardinality classes

I have written a Naive Bayes classifier (with Laplace smoothing) and am using it to classify text into a few simple classes. However, I found that the classes are not of the same vocab size -- to be ...
1
vote
0answers
27 views

Feature space reduction for tag prediction

[x-post] from stackoverflow. I am writing a ML module (python) to predict tags for a stackoverflow question (tag + body). My corpus is of around 5 million questions with title, body and tags for ...
0
votes
0answers
20 views

What algorithm can give me a probability-ranking on text-classifaction problems

I'm trying to classify e-mails using tokens in the text and headers. What I would like to know is which class a new e-mail most likely belongs to. The answer could be more than one class or could ...
0
votes
0answers
10 views

parsing semi-structured textual data poorly matrix formatted

I have found many approaches in the literature that could deal with my problem (which is parsing poorly formatted textual data having a matrix structure but whose content may vary, headers being ...
1
vote
0answers
26 views

Topic models (LDA), word cooccurances in documents?

I have read on papers that Latent Dirichlet Allocation (LDA) works by identifying word cooccurances in documents. What is confusing me is since LDA uses bag-of-words approach for document ...
1
vote
2answers
162 views

Python vs R for Text Mining Preprocessing

I've been reading some articles on cleaning text data before doing text mining analysis on it. I have experience in both Python and R and am wondering if one of these languages is an obviously better ...
0
votes
0answers
21 views

Normalization of Naive Bayes output

In Scikit-learn documentation it is possible to see that the MultinomialNB estimator has a method called predict-proba in which it has the following description: "Returns the probability of the ...
2
votes
3answers
52 views

A good intro to computational linguistics?

I have a pretty good background in data analysis and statistics in the social sciences, including both frequentist and Bayesian paradigms, and I have recently been introduced to computational ...
3
votes
2answers
109 views

What does “Virgin Data” mean?

I am using RTextTools, which has a function to create container with following syntax: create_container(matrix, labels, trainSize=NULL, testSize=NULL, virgin) ...
0
votes
1answer
107 views

Generating text data for training for doing named entity recognition and extraction

I'm trying to build an algorithm for doing named entity extraction. It goes like this. There is a large set of text documents [communications], from which specific information has to be extracted. The ...
1
vote
1answer
73 views

Text Classification using TfIdf and Bernoulli NB

So, as I am reading about Bernoulli distribution and text classification, I want to understand how Bernoulli uses TfIdf features? Since TfIdf values are within [0-1) but Multivariate Bernoulli assumes ...
0
votes
0answers
29 views

How to find the perplexity of a corpus

The formula of the perplexity measure is thus: $ p: \left(\frac{1}{\sqrt[n]{p(w_1^n)}}\right) $ where: $p(w_1^n)$ is: $\prod_{i=1}^n p(w_i)$ If I understand it correctly, this means that I ...
0
votes
1answer
51 views

A single document as input to LDA?

We use topic modelling usually on a collection of documents - which makes the input. But what if I only have a single document where I want to see the underlying topics in it? I have heard that you ...
0
votes
1answer
24 views

Multiple labels in supervised learning algorithm

I have a corpus of text with a corresponding topics. For example "A rapper Tupac was shot in LA" and it was labelled as ...
2
votes
1answer
25 views

determining significance of term use

Thing one: feel free to RTFM me: I'm definitely looking for search-able terms or background reading. Our situation is this: we have a set of 140 reviewers and 20 elements. Each reviewer reviews each ...
0
votes
0answers
22 views

The best algorithm for short documents clustering

I have a corpus of short text documents. Each document is an automatic recognized phone conversation (a dialog) from a large call center. The texts are not clean and have lots of grammar and other ...
5
votes
3answers
281 views

How would you categorize / extract information out of job descriptions?

I have a bunch of job descriptions entered by users. There are all sort of misspells and bad data. i.e: ...
2
votes
2answers
83 views

How to prepare a dataset for text classification

I would like to compare some algorithms for performing sentiment classification (Naive Bayes, SVM, and ...
0
votes
0answers
43 views

VW multiclass classification

I am new to vw and trying to do a multiclass text classification with 18 classes. features are unigram, bigram and trigrams. Total features are around 1.4 million Total training examples 35 million ...
2
votes
0answers
16 views

How to measure how 'well' I am matching Google keywords?

For google keywords you can bid on a broad match. For example let's say I bid on the keyword 'best hamburger' and somebody searches 'What sort of beef makes the best hamburger?' and 'eat best ...
0
votes
1answer
64 views

In natural language processing (NLP), how do you make an efficient dimension reduction?

In NLP, it's always the case that the dimension of the features are very huge. For example, for one project at hand, the dimension of features is almost 20 thousands (p = 20,000), and each feature is ...
1
vote
0answers
13 views

Calculate association between text documents [duplicate]

I've got 6000 reports. For each report, I've got a vector with the keywords in it and a vector with the tokens of its abstract. Now I want to calculate some association between those two sets. Is ...
1
vote
1answer
49 views

Vector Space Model for Online News Clustering

I am trying to automatically cluster news articles based on their content. I need this algorithm to be online and simply group news articles related to the same story as they arrive. The common ...
2
votes
0answers
103 views

Has the reported state-of-the-art performance of using paragraph vectors for sentiment analysis been replicated?

I was impressed by the results in the ICML 2014 paper "Distributed Representations of Sentences and Documents" by Le and Mikolov. The technique they describe, called "paragraph vectors", learns ...
0
votes
1answer
19 views

Easy way of indicating the likelihood of a text?

I am trying to find out a simple way of quantifying "how much sense a text paragraph makes". The measure is not necessarily fail-proof, but it has to be somewhat simple. One example measure that ...
3
votes
1answer
76 views

Methodology for standardizing names

We have a great amount of data with user generated names for company names (think of bills where the company wrote their own name). In our data cleaning phase, we clustered the different companies ...
0
votes
0answers
27 views

Dataset with mixed structured/unstructured data

I am looking for some mixed-type dataset with free-form textual features along with some structural features (preferably continuous) - labeled, for supervised setting (classification or prediction). ...
1
vote
2answers
109 views

How to transform test set to the PCA space of the training set, if the features in train and test are different?

I'm working on a text classification project, and I want to reduce the tf-idf matrix dimension with Principal Component Analysis (PCA) and then train my model with this, which is pretty ...
3
votes
0answers
37 views

TF-IDF Matrix and Regression

I am trying to build a regression model based on some tweets that my company put on our company feed. I would like to transform all of the tweets, and use them to tell me which word(s) were most ...
1
vote
0answers
19 views

Is Social Network Analysis or NER the best way to create a semantic graph?

I am planning to create a semantic graph by creating an automatic ontology. I want to know which is the best process to do it. Doing social network analysis to create people, relationships, likes, ...
1
vote
0answers
115 views

Any advice on how to improve my accuracy rate in text classification?

I'm trying to do a text classification task. Here are some specs: Context file size = 1M+ documents already labeled Number of top-labels = 17 Number of sub-labels = around 130 Each document is ...
0
votes
0answers
25 views

Latent Semantic Analysis: scale representation of documents?

After performing SVD on the term-document matrix, the right eigenvectors correspond to the representation of documents in the reduced concept space. In order to use this for say text classification, ...
0
votes
1answer
424 views

How to use k-fold cross validation in naive bayes classifier?

I'm trying to classify text using naive bayes classifier, and also want to use k-fold cross validation to validate the result of classification. But I'm still confused how to use the k-fold cross ...
0
votes
0answers
58 views

R Clustering Evaluation (Adaptive Kmeans)

i know there are several threads about this topic, but most i read, most i get confused. I'm doing a project that consists in clustering some data (news articles). I used adaptive Kmeans ...
3
votes
1answer
151 views

can we generate a random words from English letters that follow the bigram of the English language

The main issue is that several research building their solution of detecting and classifying English language is based on bigram distribution. However, I would like to know if it possible to generate ...
1
vote
0answers
88 views

In Kneser-Ney smoothing, how are unseen words handled?

From what I have seen, the (second-order) Kneser-Ney smoothing formula is in some way or another given as $ \begin{align} P^2_{KN}(w_n|w_{n-1}) &= \frac{\max \left\{ C\left(w_{n-1}, w_n\right) - ...
1
vote
0answers
166 views

Naive Bayes and text classification: which probability model and vectorizer combination makes sense?

I am wondering which combinations of Naive models can be paired with different vectorizing methods so that it makes sense. Let's say we have a simple binary spam-classification task. Multinomial ...
0
votes
0answers
29 views

choosing best value for N when using N-Gram approach

the question is quite general, but I am doing a research related to supervised machine learning to classify two set of characters into two categories. in fact, I want to compute some measures of ...
1
vote
1answer
220 views

Using topic words generated by LDA to represent a document

I want to do document classification by representing each document as a set of features. I know that there are many ways: BOW, TFIDF, ... I want to use Latent Dirichlet Allocation (LDA) to extract ...