Refers to a subset of data mining concerned with extracting information from data in the form of text by recognizing patterns. The goal of text mining is often to classify a given document into one of a number of categories in an automatic way, and to improve this performance dynamically, making it ...

learn more… | top users | synonyms

0
votes
0answers
9 views

TF-IDF between two different datasets in R [on hold]

I have two different datasets: one containing a product id and a query(text) and another contaning a product id and a product title. I want to calculate the tf-IDF between each of the queries and the ...
0
votes
0answers
7 views

Document Length Normalization

What is the difference between document length (total no of words in the document) normalization and cosine normalization? How do we usually normalize a document represented as a set of words and ...
0
votes
0answers
14 views

How to pre-process data & tune hyperparameters of Doc2Vec?

I am using doc2vec for getting document similarity (unsupervised learning). I read that we need to shuffle the input matrix to doc2vec & reduce the learning rate for better performance. But the ...
0
votes
0answers
15 views

How to processing a document to extract the description of certain properties of a reference domain?

I should analyze a text in order to identify the description of certain properties of some objects belonging to a given reference domain. The objects and their properties are known, as well as the ...
1
vote
0answers
38 views

Properties of Levenshtein, N-Gram, cosine and Jaccard distance coefficients - in sentence matching

Let's say I have two strings: string A: 'I went to the cafeteria and bought a sandwich.' string B: 'I heard the cafeteria is serving roast-beef sandwiches today'. ...
0
votes
0answers
13 views

Simple English translations regarding thematic textual analysis

I have run a thematic textual analysis of economic select committee meetings in the UK using t-lab. I ran a thematic analysis of the E.C.U.s, and categorised the E.C.U.s into (for example) 5 thematic ...
0
votes
0answers
7 views

Text classification using sentiment scores and test content?

Suppose there are are two snippets: t1: "Hey! I like this bag 'A' but I hope its color was better." t2: "I like most of colors the bag A comes in." You see that both statements talk about the same ...
1
vote
1answer
11 views

Choosing cases for assisted supervised learning

I have a bag of words binary text classification task. The SGD algorithm performed well for a certain target where number of labeled cases for training reached tens of thousands. For another target ...
1
vote
0answers
12 views

Ordering tdm in specific order, not frequencies

I have this corpus of documents, properly pre-processed. Working with R package tm, i do: ...
0
votes
0answers
23 views

Normalized term frequency comparisons across documents of differing length & language

I aim to infer on the prevalence of terms across and within corpora of different languages (where document length varies within and across corpora). Given Zipf’s and Heap’s laws a simple tf/n seems ...
0
votes
0answers
16 views

Which Deep Learning architecture should I use for generating one summary from different texts that talk about the same topic?

In my problem I want to develop a system capable to generate a summary or fusion of different texts that talk about similar topics (e.g. news articles). I have read about deep learning and for now it ...
0
votes
1answer
24 views

number of features in feature selection for text mining problems

Let's say for a text mining problem (e.g creating a predictive model using text analysis), using a feature selection method (e.g TF-IDF) we come up with 1000 features/words/tokens. Is there some ...
-1
votes
0answers
29 views

Statistical measure for tf.Idf weight in document

I have 100 text document with different content size. I would like to label each document using the tf.idf weight. I have calculated tf.idf for the terms in each document. I plan to give the ...
0
votes
0answers
35 views

Latent Dirichlet Allocation for Twitter dataset

I'm trying to understand Latent Dirichlet Allocation (LDA) to apply on Twitter dataset. I've a dataset with 10k tweets and I've already splitted tweets in six groups. Now I'd extract topic from each ...
0
votes
0answers
26 views

How to interpret the results of clustering on text documents

I am working on Text Analysis of the Feedback's given in a Survey. I wanted to identify the different themes or topics people are talking about. So, i have desired to go ahead and do Clustering. ...
1
vote
0answers
12 views

Data analysis approach for structured data [closed]

I have large chunk of structured data. It is mainly issues/bugs, requirements submitted on a web, stored in a database. e.g. data will be organized as Title, ID, Subject detail, issue ingredient, ...
1
vote
0answers
34 views

Where did sublinear tf-idf originate?

I have often come across this weighting scheme for tf-idf (term frequency - inverse document frequency) in text mining. I am wondering where it came from (for citations). I've searched very rigorously,...
0
votes
0answers
25 views

Linear model coefficients with text mining

Suppose that I have a collection of reviews about food - perhaps reviews for a restaurant or something. Also suppose that I'm interested in predicting the score from these reviews. One approach that I ...
0
votes
1answer
24 views

TF IDF Analysis for tag extraction

So I have used the tm package to perform TF IDF analysis on around 1000 documents. What I want to do is to extract common tags from those documents. At the moment I already have a matrix where the ...
0
votes
0answers
32 views

Why getting better classification results despite many irrelevant terms?

I am new to ML especially for document (text) classification. I have 22 classes (scientific fields) and I am trying to improve classification results by employing some additional data. That is, I use ...
1
vote
0answers
46 views

Pros and Cons: LDA vs Neural Networks

LDA is an older approach for word representations, there are newer methods now like CBOW and Skip-gram. But what are the improvements of these models? Do they improve in every way or does LDA still ...
1
vote
0answers
28 views

Selecting correct settings for the order of Minkowski distance

I am looking to compute the distance between vectors of word frequencies (and I am new to this). I am trying out the Minkowski distance as implemented in Scipy. The documentation asks me to specify a "...
0
votes
0answers
20 views

How large should be negative train set in text classification of rare category

I'm working on the classification of medical texts in order to find texts about the quite rare disease from the big set of all medical articles. I have the set of positive examples and some negative ...
1
vote
0answers
18 views

Getting significantly different results for a classification task when using two similar approaches

I am new to machine learning and I classify abstracts of scientific papers retrieved from two different disciplines. I use RTextTools package and I apply two different approaches for the ...
0
votes
0answers
27 views

Text mining to find the significant date in a news article

Lets say we have a group of news articles that have already been classified as pertaining to an event (such as a conference or public announcment). The last step in the problem is to determine what ...
0
votes
0answers
30 views

What is the meaning of laplace, eps and threshold in NaiveBayes package in R e1071 lib?

I am using NaiveBayes for text classification, I am interested on tagging a text (like a blog post). What I am finding is that normally I have results in which a tag has a probability of 0.9999 of ...
0
votes
0answers
16 views

How to assess text classification results when extreme sparsity is present?

I try to classify documents based on bag-of-words single word approach. I employ R with its RTextTools to use SVM. My text files are like below: TRAIN ...
1
vote
2answers
76 views

How to use MaxEnt (logistic regression) weights?

I asked this question last night and Matt Krause explanation helped me a lot. (For more explanation please see my previous question). Now I have another problem. We are using RapidMiner studio and ...
1
vote
1answer
60 views

Low recall and high precision in text summarization

We are trying to generate a model to summarize Persian news. About 14000 news were summarized with help of humans(supervised) and then we extracted all sentences (about 180000) and labeled them (...
1
vote
1answer
32 views

Latent Semantic Analysis: stop words and link words

On many tutorials about how to implement LSA, I see that stop words such as "and" are removed. I understand that we might find them in almost all kind of texts, but the repetition of link words in a ...
0
votes
0answers
9 views

Train Textual Pattern Detector

I'm trying to train a model that can detect structures of text (and maybe label them in case of multiple structures) in a coprus. Exemple: Train a model with a dataset of addresses: ...
0
votes
0answers
25 views

text mining (document term matrix)

I am doing a text mining project in R using "tm" package . I have successfully built a document term matrix. I want to remove terms having frequency >75 % and terms having frequency <25 % ...
1
vote
1answer
182 views

What is text distance in data mining

I need to write a report on visualization of multidimensional data, map and text distance. I got content related to other two but not getting any clue about text distance. Is it related to Data ...
1
vote
1answer
33 views

Text classification algorithms for small sets

I'm trying to classify a set of 1656 tweets into different categories. I've read about different classification algorithms (supervised and unsupervised) but I'm really concerned because my set and ...
1
vote
0answers
18 views

Is there any package in R for Sound like analysys of text [closed]

The words "Jhon" and "Joan" may sound similar although spelling is different. Is there any package for "Sound like Analysis" in R
0
votes
1answer
30 views

Doubt about feature selection

I'm working on a text classification problem using Python and NLTK. I've got two frequency distributions, one for each class (it's basically a binary classification). So, my doubt it's if there's a ...
3
votes
0answers
25 views

How to do postal addresses fuzzy matching?

I would like to know how to match postal addresses when their format differ or when one of them is mispelled. So far I've found different solutions but I think that they are quite old and not very ...
1
vote
0answers
19 views

PCA: entire feature space or grouped

Ok let me explain the question: i'm doing text classification using standard tfidf transformations. It happens that my 'corpus' can be divided into groups, for example: 'words in title', 'words in ...
0
votes
0answers
16 views

Neural Network Structure Sentence

I'm new in Stats SE. I'm trying to figure how can I can give a preprocessed sentence (with dependency parsing structure and pos tags), and prepare a training set, to my network be able to predict the ...
3
votes
1answer
70 views

Text Clustering using TF-IDF and Cosine Similarity

I am attempting to perform hierarchical clustering using (Tf-Idf & cosine distance) on about 25,000 documents that vary in length between 1-3 paragraphs each. With the method above, my question ...
0
votes
0answers
42 views

Reinforcement Learning & Text Mining

I was wondering if one could use Reinforcement Learning (as it is going to be more and more trendy with the DeepMind & AlphaGo's stuff) to parse and extract information from text. For example, ...
0
votes
0answers
33 views

text mining cross validation / leave one out

I'm having a small data set for text mining classification task (pos vs neg). The process consists of building the document term matrix(DTM) and then train an svm. If I'm making the train either with ...
0
votes
0answers
35 views

SGDClassifier shows poor performance on larger dataset, what to check?

I get a total loss of precision moving from UK to US dataset (80K to 1M subjects). What could go wrong? I have successfully applied SGDClassifier for binary classification on data-sets with as many ...
2
votes
0answers
45 views

Cross Co-occurrence between two corpora

I've looked around for a solution to this problem specifically in nltk, quite a bit but couldn't find much help either on SO or elsewhere. My problem is as follows: I have a set of aligned pairs of ...
0
votes
0answers
40 views

Text Mining Using irlba for dimension reduction

I am trying to do some dimension reduction on a sparse matrix I have. The data is text data currently formatted in a document term matrix. I did some reading and used the irlba package to reduce the ...
0
votes
0answers
14 views

Gaming the ROUGE metric for text summarization

ROUGE seems to be the standard way of evaluating the quality of machine generated summaries of text documents by comparing them with reference summaries (human generated). $$ROUGE_{n}= \frac {\sum_{s\...
0
votes
0answers
8 views

Maximum vocabulary distance

Given a vocabulary with size m (the number of letters in it) and words of length n, what is the maximum word distance (number of differing letters) for a text with length o (the number of words in it)?...
0
votes
2answers
78 views

Highly correlated features in text mining despite mutual information criterion

I'm trying to classify documents into two classes using the Bernoulli Naive Bayes algorithm, as described here in chapter 13. I've extracted 500 tokens (out of more than 30,000) from my sample ...
0
votes
0answers
9 views

TF-IDF for matching 2 titles?

My question can look irrelevant. But I guess, it's better to ask here, rather than on StackOverflow. Let's consume, we have 2 long titles, like: ...
1
vote
0answers
14 views

Finding cross-disciplinary themes in course listings

I’m building a dataset of course listings and am looking for methods/ algorithms to reveal clusters, connections, cross-disciplinary themes and threads that could be found in different courses. ...