Refers to a subset of data mining concerned with extracting information from data in the form of text by recognizing patterns. The goal of text mining is often to classify a given document into one of a number of categories in an automatic way, and to improve this performance dynamically, making it ...

learn more… | top users | synonyms

0
votes
0answers
16 views

Why getting better classification results despite many irrelevant terms?

I am new to ML especially for document (text) classification. I have 22 classes (scientific fields) and I am trying to improve classification results by employing some additional data. That is, I use ...
0
votes
0answers
9 views

Problem parsing strings with grep/str_extract [migrated]

As part of my feature engineering, I need to parse text strings from different languages and keep text enclosed within parentheses. Everything was going well until I encountered a very strange ...
-1
votes
0answers
13 views

Word relevance on a topic from different documents/message

I have a set of texts in which lines of text are tagged with 1 & 0. I want to know which words are most likely to occur on ...
0
votes
0answers
62 views

Sentiment analysis, text analysis, data mining unstructured data [on hold]

What is the best API/tool that can be used in c# to make sense of unstructured data communication to be interpreted for business use? Thanks for your help in advance.
0
votes
0answers
17 views

Clustering words by similarity and occurance

I have the following problem: I have a list of document terms and their frequency. These terms are not common English words. Numerous variants (i.e. spelling mistakes) relating to these terms are ...
1
vote
0answers
24 views

Pros and Cons: LDA vs Neural Networks

LDA is an older approach for word representations, there are newer methods now like CBOW and Skip-gram. But what are the improvements of these models? Do they improve in every way or does LDA still ...
1
vote
0answers
18 views

Selecting correct settings for the order of Minkowski distance

I am looking to compute the distance between vectors of word frequencies (and I am new to this). I am trying out the Minkowski distance as implemented in Scipy. The documentation asks me to specify a ...
0
votes
0answers
16 views

How large should be negative train set in text classification of rare category

I'm working on the classification of medical texts in order to find texts about the quite rare disease from the big set of all medical articles. I have the set of positive examples and some negative ...
1
vote
0answers
13 views

Getting significantly different results for a classification task when using two similar approaches

I am new to machine learning and I classify abstracts of scientific papers retrieved from two different disciplines. I use RTextTools package and I apply two different approaches for the ...
0
votes
0answers
24 views

Text mining to find the significant date in a news article

Lets say we have a group of news articles that have already been classified as pertaining to an event (such as a conference or public announcment). The last step in the problem is to determine what ...
0
votes
0answers
12 views

What is the meaning of laplace, eps and threshold in NaiveBayes package in R e1071 lib?

I am using NaiveBayes for text classification, I am interested on tagging a text (like a blog post). What I am finding is that normally I have results in which a tag has a probability of 0.9999 of ...
0
votes
0answers
13 views

How to assess text classification results when extreme sparsity is present?

I try to classify documents based on bag-of-words single word approach. I employ R with its RTextTools to use SVM. My text files are like below: TRAIN ...
1
vote
2answers
61 views

How to use MaxEnt (logistic regression) weights?

I asked this question last night and Matt Krause explanation helped me a lot. (For more explanation please see my previous question). Now I have another problem. We are using RapidMiner studio and ...
1
vote
1answer
47 views

Low recall and high precision in text summarization

We are trying to generate a model to summarize Persian news. About 14000 news were summarized with help of humans(supervised) and then we extracted all sentences (about 180000) and labeled them (...
1
vote
1answer
27 views

Latent Semantic Analysis: stop words and link words

On many tutorials about how to implement LSA, I see that stop words such as "and" are removed. I understand that we might find them in almost all kind of texts, but the repetition of link words in a ...
0
votes
0answers
7 views

Train Textual Pattern Detector

I'm trying to train a model that can detect structures of text (and maybe label them in case of multiple structures) in a coprus. Exemple: Train a model with a dataset of addresses: ...
0
votes
0answers
14 views

text mining (document term matrix)

I am doing a text mining project in R using "tm" package . I have successfully built a document term matrix. I want to remove terms having frequency >75 % and terms having frequency <25 % ...
1
vote
1answer
148 views

What is text distance in data mining

I need to write a report on visualization of multidimensional data, map and text distance. I got content related to other two but not getting any clue about text distance. Is it related to Data ...
1
vote
1answer
28 views

Text classification algorithms for small sets

I'm trying to classify a set of 1656 tweets into different categories. I've read about different classification algorithms (supervised and unsupervised) but I'm really concerned because my set and ...
1
vote
0answers
16 views

Is there any package in R for Sound like analysys of text [closed]

The words "Jhon" and "Joan" may sound similar although spelling is different. Is there any package for "Sound like Analysis" in R
0
votes
1answer
25 views

Doubt about feature selection

I'm working on a text classification problem using Python and NLTK. I've got two frequency distributions, one for each class (it's basically a binary classification). So, my doubt it's if there's a ...
3
votes
0answers
19 views

How to do postal addresses fuzzy matching?

I would like to know how to match postal addresses when their format differ or when one of them is mispelled. So far I've found different solutions but I think that they are quite old and not very ...
1
vote
0answers
16 views

PCA: entire feature space or grouped

Ok let me explain the question: i'm doing text classification using standard tfidf transformations. It happens that my 'corpus' can be divided into groups, for example: 'words in title', 'words in ...
0
votes
0answers
13 views

Neural Network Structure Sentence

I'm new in Stats SE. I'm trying to figure how can I can give a preprocessed sentence (with dependency parsing structure and pos tags), and prepare a training set, to my network be able to predict the ...
3
votes
1answer
36 views

Text Clustering using TF-IDF and Cosine Similarity

I am attempting to perform hierarchical clustering using (Tf-Idf & cosine distance) on about 25,000 documents that vary in length between 1-3 paragraphs each. With the method above, my question ...
0
votes
0answers
31 views

Reinforcement Learning & Text Mining

I was wondering if one could use Reinforcement Learning (as it is going to be more and more trendy with the DeepMind & AlphaGo's stuff) to parse and extract information from text. For example, ...
0
votes
0answers
25 views

text mining cross validation / leave one out

I'm having a small data set for text mining classification task (pos vs neg). The process consists of building the document term matrix(DTM) and then train an svm. If I'm making the train either with ...
0
votes
0answers
19 views

SGDClassifier shows poor performance on larger dataset, what to check?

I get a total loss of precision moving from UK to US dataset (80K to 1M subjects). What could go wrong? I have successfully applied SGDClassifier for binary classification on data-sets with as many ...
1
vote
0answers
21 views

Cross Co-occurrence between two corpora

I've looked around for a solution to this problem specifically in nltk, quite a bit but couldn't find much help either on SO or elsewhere. My problem is as follows: I have a set of aligned pairs of ...
0
votes
0answers
31 views

Text Mining Using irlba for dimension reduction

I am trying to do some dimension reduction on a sparse matrix I have. The data is text data currently formatted in a document term matrix. I did some reading and used the irlba package to reduce the ...
0
votes
0answers
6 views

Gaming the ROUGE metric for text summarization

ROUGE seems to be the standard way of evaluating the quality of machine generated summaries of text documents by comparing them with reference summaries (human generated). $$ROUGE_{n}= \frac ...
0
votes
0answers
7 views

Maximum vocabulary distance

Given a vocabulary with size m (the number of letters in it) and words of length n, what is the maximum word distance (number of differing letters) for a text with length o (the number of words in ...
0
votes
2answers
71 views

Highly correlated features in text mining despite mutual information criterion

I'm trying to classify documents into two classes using the Bernoulli Naive Bayes algorithm, as described here in chapter 13. I've extracted 500 tokens (out of more than 30,000) from my sample ...
0
votes
0answers
6 views

TF-IDF for matching 2 titles?

My question can look irrelevant. But I guess, it's better to ask here, rather than on StackOverflow. Let's consume, we have 2 long titles, like: ...
1
vote
0answers
14 views

Finding cross-disciplinary themes in course listings

I’m building a dataset of course listings and am looking for methods/ algorithms to reveal clusters, connections, cross-disciplinary themes and threads that could be found in different courses. ...
2
votes
2answers
41 views

Text categorization using Naive Bayes: Why isn't this working?

I'm trying to implement a system for text categorization using Naive Bayes as part of a school project. I have to hand code the algorithm and have been having some issues. To make sure I understand ...
0
votes
0answers
23 views

How does TextRank differentiate between keywords and simply frequent words, such as “is”

I am trying to understand how TextRank document summary algorithm works. A few articles that I've read so far introduce text rank as a modification of page rank (e.g. article in wikipedia). However, ...
0
votes
1answer
71 views

Interpreting Cluster Analysis from SAS Enterprise Miner

I am currently doing a text mining project and I conducted a clustering analysis in SAS enterprise miner. I am using the following settings: Anyway, The results look like this, showing me ...
0
votes
0answers
35 views

How to transform the LDA model to see the topic evolution in chat content?

Now I have a data set with about 13,000 lines, including the date, sender, chat content in a public server. The data set covers about ...
1
vote
0answers
36 views

Minimum Spanning Tree in R (vegan): managing identical values

In R the package "vegan" contains the function spantree. It takes a matrix of distances among terms and it creates a tree with all the points... unless two or more rows (ex: A, B, C) are identical. ...
0
votes
0answers
41 views

Calculate similarity of probability vector and TFIDF vector

I want to compute the similarity between documents of two groups of documents. All documents have the same term vector of length 100, but documents in group A are probabilities, whereas documents in ...
0
votes
0answers
19 views

Text Classification Issues

In text mining, there are too many variables (features) and the features have multiple permutations. E.g. best, not best, not the best, one of the best, bestest, may not be the best etc. creating ...
0
votes
0answers
15 views

Text Mining - Reports Optimization

I am working on optimizing ~8000 financial and operational reports which have frequency ranging from monthly, quarterly and yearly. To accomplish this I am using text mining to identify similar and ...
0
votes
0answers
40 views

How do you compare words or documents using LSA (latent semantic analysis)

As the title says, I am a bit cofunsed in how documents or words are compared using LSA (when I say compare, I am referring to calculate similarities, for instance, cosine similarity). In An ...
0
votes
0answers
13 views

Content Based Document classification

I have a corpus of 10 million resumes. I want to add tags to these resumes like Software Engineer, Data Scientists, ...
1
vote
0answers
92 views

calculating PMI for co-occurrences of words

I am in the process of building a question answering system. I am interested in calculating the PMI for words $x$ and $y$ occurring within 5 words of each other in a document. I have the formula and ...
0
votes
0answers
16 views

How do I mine text data for a list of “unspecified keywords” from a bunch of documents?

I need analytics on "skills" from a bunch of documents. The straightforward way for me is to first create a dictionary of "skills keywords" and then get descriptive analytics from the documents i.e. ...
0
votes
1answer
73 views

Text Classification for (Short) Open-Ended Survey Responses?

I am new to text mining/classification, and really want to learn more from this community. My data are open-ended survey responses (n=about 6,000) in which the respondent described what happened at ...
0
votes
0answers
12 views

Comparing two annotated datasets

My task is to compare annotation quality of two datasets, one with respect to another. The datasets have been generated by two distinct groups of people, one is believed to be more ...