Refers to a subset of data mining concerned with extracting information from data in the form of text by recognizing patterns. The goal of text mining is often to classify a given document into one of a number of categories in an automatic way, and to improve this performance dynamically, making it ...

learn more… | top users | synonyms

0
votes
0answers
16 views

Latent Dirichlet Allocation for Twitter dataset

I'm trying to understand Latent Dirichlet Allocation (LDA) to apply on Twitter dataset. I've a dataset with 10k tweets and I've already splitted tweets in six groups. Now I'd extract topic from each ...
0
votes
0answers
16 views

How to interpret the results of clustering on text documents

I am working on Text Analysis of the Feedback's given in a Survey. I wanted to identify the different themes or topics people are talking about. So, i have desired to go ahead and do Clustering. ...
1
vote
0answers
11 views

Data analysis approach for structured data [closed]

I have large chunk of structured data. It is mainly issues/bugs, requirements submitted on a web, stored in a database. e.g. data will be organized as Title, ID, Subject detail, issue ingredient, ...
1
vote
0answers
31 views

Where did sublinear tf-idf originate?

I have often come across this weighting scheme for tf-idf (term frequency - inverse document frequency) in text mining. I am wondering where it came from (for citations). I've searched very ...
0
votes
0answers
24 views

Linear model coefficients with text mining

Suppose that I have a collection of reviews about food - perhaps reviews for a restaurant or something. Also suppose that I'm interested in predicting the score from these reviews. One approach that I ...
0
votes
1answer
22 views

TF IDF Analysis for tag extraction

So I have used the tm package to perform TF IDF analysis on around 1000 documents. What I want to do is to extract common tags from those documents. At the moment I already have a matrix where the ...
0
votes
0answers
29 views

Why getting better classification results despite many irrelevant terms?

I am new to ML especially for document (text) classification. I have 22 classes (scientific fields) and I am trying to improve classification results by employing some additional data. That is, I use ...
0
votes
0answers
9 views

Problem parsing strings with grep/str_extract [migrated]

As part of my feature engineering, I need to parse text strings from different languages and keep text enclosed within parentheses. Everything was going well until I encountered a very strange ...
-1
votes
0answers
14 views

Word relevance on a topic from different documents/message

I have a set of texts in which lines of text are tagged with 1 & 0. I want to know which words are most likely to occur on ...
1
vote
0answers
37 views

Pros and Cons: LDA vs Neural Networks

LDA is an older approach for word representations, there are newer methods now like CBOW and Skip-gram. But what are the improvements of these models? Do they improve in every way or does LDA still ...
1
vote
0answers
23 views

Selecting correct settings for the order of Minkowski distance

I am looking to compute the distance between vectors of word frequencies (and I am new to this). I am trying out the Minkowski distance as implemented in Scipy. The documentation asks me to specify a ...
0
votes
0answers
19 views

How large should be negative train set in text classification of rare category

I'm working on the classification of medical texts in order to find texts about the quite rare disease from the big set of all medical articles. I have the set of positive examples and some negative ...
1
vote
0answers
17 views

Getting significantly different results for a classification task when using two similar approaches

I am new to machine learning and I classify abstracts of scientific papers retrieved from two different disciplines. I use RTextTools package and I apply two different approaches for the ...
0
votes
0answers
26 views

Text mining to find the significant date in a news article

Lets say we have a group of news articles that have already been classified as pertaining to an event (such as a conference or public announcment). The last step in the problem is to determine what ...
0
votes
0answers
24 views

What is the meaning of laplace, eps and threshold in NaiveBayes package in R e1071 lib?

I am using NaiveBayes for text classification, I am interested on tagging a text (like a blog post). What I am finding is that normally I have results in which a tag has a probability of 0.9999 of ...
0
votes
0answers
15 views

How to assess text classification results when extreme sparsity is present?

I try to classify documents based on bag-of-words single word approach. I employ R with its RTextTools to use SVM. My text files are like below: TRAIN ...
1
vote
2answers
66 views

How to use MaxEnt (logistic regression) weights?

I asked this question last night and Matt Krause explanation helped me a lot. (For more explanation please see my previous question). Now I have another problem. We are using RapidMiner studio and ...
1
vote
1answer
52 views

Low recall and high precision in text summarization

We are trying to generate a model to summarize Persian news. About 14000 news were summarized with help of humans(supervised) and then we extracted all sentences (about 180000) and labeled them (...
1
vote
1answer
28 views

Latent Semantic Analysis: stop words and link words

On many tutorials about how to implement LSA, I see that stop words such as "and" are removed. I understand that we might find them in almost all kind of texts, but the repetition of link words in a ...
0
votes
0answers
8 views

Train Textual Pattern Detector

I'm trying to train a model that can detect structures of text (and maybe label them in case of multiple structures) in a coprus. Exemple: Train a model with a dataset of addresses: ...
0
votes
0answers
23 views

text mining (document term matrix)

I am doing a text mining project in R using "tm" package . I have successfully built a document term matrix. I want to remove terms having frequency >75 % and terms having frequency <25 % ...
1
vote
1answer
159 views

What is text distance in data mining

I need to write a report on visualization of multidimensional data, map and text distance. I got content related to other two but not getting any clue about text distance. Is it related to Data ...
1
vote
1answer
30 views

Text classification algorithms for small sets

I'm trying to classify a set of 1656 tweets into different categories. I've read about different classification algorithms (supervised and unsupervised) but I'm really concerned because my set and ...
1
vote
0answers
16 views

Is there any package in R for Sound like analysys of text [closed]

The words "Jhon" and "Joan" may sound similar although spelling is different. Is there any package for "Sound like Analysis" in R
0
votes
1answer
28 views

Doubt about feature selection

I'm working on a text classification problem using Python and NLTK. I've got two frequency distributions, one for each class (it's basically a binary classification). So, my doubt it's if there's a ...
3
votes
0answers
23 views

How to do postal addresses fuzzy matching?

I would like to know how to match postal addresses when their format differ or when one of them is mispelled. So far I've found different solutions but I think that they are quite old and not very ...
1
vote
0answers
18 views

PCA: entire feature space or grouped

Ok let me explain the question: i'm doing text classification using standard tfidf transformations. It happens that my 'corpus' can be divided into groups, for example: 'words in title', 'words in ...
0
votes
0answers
14 views

Neural Network Structure Sentence

I'm new in Stats SE. I'm trying to figure how can I can give a preprocessed sentence (with dependency parsing structure and pos tags), and prepare a training set, to my network be able to predict the ...
3
votes
1answer
53 views

Text Clustering using TF-IDF and Cosine Similarity

I am attempting to perform hierarchical clustering using (Tf-Idf & cosine distance) on about 25,000 documents that vary in length between 1-3 paragraphs each. With the method above, my question ...
0
votes
0answers
36 views

Reinforcement Learning & Text Mining

I was wondering if one could use Reinforcement Learning (as it is going to be more and more trendy with the DeepMind & AlphaGo's stuff) to parse and extract information from text. For example, ...
0
votes
0answers
31 views

text mining cross validation / leave one out

I'm having a small data set for text mining classification task (pos vs neg). The process consists of building the document term matrix(DTM) and then train an svm. If I'm making the train either with ...
0
votes
0answers
27 views

SGDClassifier shows poor performance on larger dataset, what to check?

I get a total loss of precision moving from UK to US dataset (80K to 1M subjects). What could go wrong? I have successfully applied SGDClassifier for binary classification on data-sets with as many ...
1
vote
0answers
31 views

Cross Co-occurrence between two corpora

I've looked around for a solution to this problem specifically in nltk, quite a bit but couldn't find much help either on SO or elsewhere. My problem is as follows: I have a set of aligned pairs of ...
0
votes
0answers
36 views

Text Mining Using irlba for dimension reduction

I am trying to do some dimension reduction on a sparse matrix I have. The data is text data currently formatted in a document term matrix. I did some reading and used the irlba package to reduce the ...
0
votes
0answers
11 views

Gaming the ROUGE metric for text summarization

ROUGE seems to be the standard way of evaluating the quality of machine generated summaries of text documents by comparing them with reference summaries (human generated). $$ROUGE_{n}= \frac ...
0
votes
0answers
8 views

Maximum vocabulary distance

Given a vocabulary with size m (the number of letters in it) and words of length n, what is the maximum word distance (number of differing letters) for a text with length o (the number of words in ...
0
votes
2answers
75 views

Highly correlated features in text mining despite mutual information criterion

I'm trying to classify documents into two classes using the Bernoulli Naive Bayes algorithm, as described here in chapter 13. I've extracted 500 tokens (out of more than 30,000) from my sample ...
0
votes
0answers
7 views

TF-IDF for matching 2 titles?

My question can look irrelevant. But I guess, it's better to ask here, rather than on StackOverflow. Let's consume, we have 2 long titles, like: ...
1
vote
0answers
14 views

Finding cross-disciplinary themes in course listings

I’m building a dataset of course listings and am looking for methods/ algorithms to reveal clusters, connections, cross-disciplinary themes and threads that could be found in different courses. ...
2
votes
2answers
41 views

Text categorization using Naive Bayes: Why isn't this working?

I'm trying to implement a system for text categorization using Naive Bayes as part of a school project. I have to hand code the algorithm and have been having some issues. To make sure I understand ...
0
votes
0answers
25 views

How does TextRank differentiate between keywords and simply frequent words, such as “is”

I am trying to understand how TextRank document summary algorithm works. A few articles that I've read so far introduce text rank as a modification of page rank (e.g. article in wikipedia). However, ...
0
votes
1answer
110 views

Interpreting Cluster Analysis from SAS Enterprise Miner

I am currently doing a text mining project and I conducted a clustering analysis in SAS enterprise miner. I am using the following settings: Anyway, The results look like this, showing me ...
0
votes
0answers
39 views

How to transform the LDA model to see the topic evolution in chat content?

Now I have a data set with about 13,000 lines, including the date, sender, chat content in a public server. The data set covers about ...
1
vote
0answers
39 views

Minimum Spanning Tree in R (vegan): managing identical values

In R the package "vegan" contains the function spantree. It takes a matrix of distances among terms and it creates a tree with all the points... unless two or more rows (ex: A, B, C) are identical. ...
0
votes
0answers
45 views

Calculate similarity of probability vector and TFIDF vector

I want to compute the similarity between documents of two groups of documents. All documents have the same term vector of length 100, but documents in group A are probabilities, whereas documents in ...
0
votes
0answers
21 views

Text Classification Issues

In text mining, there are too many variables (features) and the features have multiple permutations. E.g. best, not best, not the best, one of the best, bestest, may not be the best etc. creating ...
0
votes
0answers
15 views

Text Mining - Reports Optimization

I am working on optimizing ~8000 financial and operational reports which have frequency ranging from monthly, quarterly and yearly. To accomplish this I am using text mining to identify similar and ...
0
votes
0answers
48 views

How do you compare words or documents using LSA (latent semantic analysis)

As the title says, I am a bit cofunsed in how documents or words are compared using LSA (when I say compare, I am referring to calculate similarities, for instance, cosine similarity). In An ...
0
votes
0answers
13 views

Content Based Document classification

I have a corpus of 10 million resumes. I want to add tags to these resumes like Software Engineer, Data Scientists, ...
1
vote
0answers
139 views

calculating PMI for co-occurrences of words

I am in the process of building a question answering system. I am interested in calculating the PMI for words $x$ and $y$ occurring within 5 words of each other in a document. I have the formula and ...