Stack Exchange Network

Stack Exchange network consists of 175 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [text-mining]

Refers to a subset of data mining concerned with extracting information from data in the form of text by recognizing patterns. The goal of text mining is often to classify a given document into one of a number of categories in an automatic way, and to improve this performance dynamically, making it an example of machine learning. One example of this type of text mining are spam filters used for email.

0
votes
0answers
14 views

Are there any data formats for storing text worth looking into, besides CSV? [closed]

I have noticed Pandas has several storage options, pickle, feather, parquet, sql, hdf5, etc. Are any of these worth looking into for simple text data? If it makes a difference, I am mostly looking ...
0
votes
0answers
16 views

binary classification problem for data where one class potentially includes members of the other class

Situation Assuming one wants to classify text data into two sentiment categories: negative and neutral (i.e everything that is not negative). For the training process, labelled data is available. ...
0
votes
0answers
20 views

Classify documents using a set of known vocabularies

I have a bunch of documents that I want to classify which ones talk about soccer (unsupervised learning, I do not want to manually label the documents). One way I am thinking about is to go online ...
0
votes
0answers
22 views

Improve the accuracy of semantic text matching

I have a corpus of ~200K sentences of variable length, the median length is 16 words. My goal is for a given sentence to find other sentences with a similar meaning. I tried several approaches: using ...
0
votes
1answer
12 views

Should I convert classification output to integer and how?

I'm using a neural network to classify text, and the label of the training data is 0 or 1(i.e. binary classification). It works well in the training and evaluating process, but the prediction output ...
0
votes
0answers
9 views

classification association rules and closed frequent itemsets

if I need to mine all frequent itemsets for text classification using classification association rules. what is the most efficient algorithm? is CHARM algorithm efficient with mining only closed ...
0
votes
0answers
33 views

Visualizing a Latent Dirichlet Allocation (LDA) by Multidimensional Scaling (MDS)

I did an LDA with four topics for four different Smartphones. This was done using customer Reviews of Amazon. ...
0
votes
0answers
4 views

Combine Noisy predictions from an optical character recognition program

I am trying to perform optical character recognition on a field of text from different angles as the camera passes over it and beyond it. Due to the 3D skewing of the image, thee readings of the OCR ...
0
votes
0answers
15 views

How to denote token length in statistics

I have a vocabulary (V) that includes terms. (e.g., V = ['data mining', 'data', 'machine', 'machine learning']) I also have a ...
0
votes
1answer
21 views

Get association of categorical variables

I do a text analysis where I want to identify dependencies among categorical variables, for example let's take this dataset: ...
0
votes
0answers
13 views

Using LDA to create two layered wordcloud

I create a word-cloud using LDA model what I want to do is to find the documents IDs related to that topic group. So, for example, the image here I want to allow users to click the word broccoli and ...
0
votes
0answers
13 views

Correlations in text analysis - for an absolute beginner

I am analysing the media releases and speeches of two political leaders. I want to find out: how often they mention young people, and in what context. I know my frequency tables are correct because I ...
0
votes
0answers
10 views

How PV-DBOW works

The authors of the Paragraph Vector paper describe PV-DBOW with: 2.3. Paragraph Vector without word ordering: Distributed bag of words The above method considers the concatenation of the ...
0
votes
1answer
50 views

How to find correlation between text data?

I have data set similar to this: I want to if the columns subtype and item are correlated. They have different text, ...
0
votes
0answers
14 views

How to solve the estimate population characters with sample text-mining?

I've recently learning text-mining, but none of my textbooks talk of inferential statistics; they talk about how to analyze collected data but hardly deal with how to estimate population data through ...
0
votes
0answers
5 views

wordfish - must reference texts be at the absolute extreme of the spectrum

I have a question pertaining to the wordfish method (see here for the academic paper introducing the package and here for the package itself). My question regards ...
0
votes
0answers
23 views

How to statistically infer common pattern in text

Am trying to solve a problem where I need to infer common patterns in text for example, the data below, with bare eyes it can be noticed there is a pattern and that is ...
0
votes
0answers
7 views

Calculating the ratio of the (frequency of a specific word in the corpus/sum of the frequency of all the words)

I have the following code which gives me the list of top 10 words in a corpus in descending order of frequency: library(tidytext) tidybooks.nstop<-tidy_books %>% + anti_join(stop_words) ...
0
votes
0answers
16 views

Classification of sample with only unseen words

I'm doing text classification (Product Name) where one example belongs to one class. "Some Product Name" -> MODEL -> {CLASS_1 | CLASS_2 | CLASS_3 | CLASS_4} ...
2
votes
0answers
73 views

How to compute gain statistic for the multinomial Naive Bayes classifier from Jurafsky and Martin (2018)

I'm trying to figure out how to compute the gain statistic G(w) following the fitting of the multinomial Naive Bayes model. This statistic is described on p17 of the new edition of Jurafsky and ...
0
votes
0answers
27 views

Sentiment Analysis Issues Using R

I am using the R package sentimentr and sentiment function to get polarity/sentiment scores on a list of comments. The issue I am having is that the comments are in ...
0
votes
0answers
19 views

Difference between Semantic analysis and Syntactic analysis in text analysis

Can someone explain me clearly the meaning of Semantic analysis and Syntactic Analysis in text analysis ? I am very much confused between these two. Also tell the difference in how they are used in ...
0
votes
0answers
17 views

How to ensemble predictions from image classifier and text classifier?

I am doing multiclass classification based on images and text. I have predictions from both image classification and text. I am not sure how to combine them. Should I use probabilities as a feature to ...
0
votes
0answers
23 views

Guided LDA to categorize software requirements

I'm developing a application to categorize requirements in a requirement specification in to categories like database, front end, back end, etc. So for that I'm trying to use Guided LDA since labelled ...
0
votes
0answers
27 views

Pointwise Mutual Information Word Dependency

I have pre-defined concepts which are either a single word or couple of words that refer to a concept.( In the context of machine learning for instance, covariance matrix is a concept). I am trying to ...
2
votes
0answers
15 views

Correlation of phrases and answers to them

If I ask question in wrong forum, let me know, I'll delete it. I try find methods, methodology to create predicative model. Namely, I want to investigate the relationship between the sentences and ...
0
votes
0answers
15 views

naive bayes text classification: can I look at the individual word probabilities?

Assume I use the Naive Bayes classification algorithm. My question is simple: can I rank the words according to their posterior probabilities? I want to have a measure of "importance" of the words in ...
1
vote
1answer
12 views

Good way to use word similarity as a feature in supervised ML on text

I have a pretty low N data set of small sentences tagged with a label. I would like to create a classifier on this dataset. The word choice is not very variable since the domain is pretty specific. ...
0
votes
0answers
21 views

Using a priori knowledge in a classification task

I'm working on a classification task, related with text classification, where texts to be classified are requests for technical support, and the classes are technical guys which issues can be assigned ...
1
vote
1answer
58 views

Finding similar text - algorithms and evaluation

I've been asked to create a program that will rank similar texts to an input text given a collection of text. So far I've been using a tdidf representation and cosine similarity with a lot of regex-...
0
votes
0answers
40 views

Getting the document-per-topic loading using TextmineR package by passing term co-occurrence matrix

I am using TextmineR package to find the most similar documents to given document list. I used the following code to generate the tcm not dtm ...
0
votes
1answer
21 views

apply CountVectorized to whole data before applying train_test_split

Is there any difference between the two different snips of codes. ...
0
votes
0answers
19 views

Required sample size for sentiment analysis in customer satisfaction

Suppose we are interested in seeing how companies A through F are being viewed by different customers, and how this changes based on the customers' positions. A sample of customers is drawn from ...
1
vote
1answer
20 views

Information theoretic alternative to tf-idf heuristic?

I've been recently working with feature construction from texts, where tf-idf measure is one of the main options for vectorizing the documents (one feature per e.g., word). I was wondering, whether ...
0
votes
0answers
17 views

What is the best model for keyphrase extraction from super long text?

I’m working on a keyphrase extraction task. The biggest difficulty of this task is that the text is very long (5000-20000 words). I’ve tried several unsupervised algorithms such as Tf-idf and TextRank ...
0
votes
1answer
463 views

Imbalanced multiclass classification with many classes

I am working on a text classification project in which we have hundreds of (imbalanced) classes. Some characteristics of the data: We have examples of "bad" documents. Basically documents that don't ...
0
votes
0answers
58 views

Applying Label From Supervised Learning to Unlabeled Data- Text Classification

I am wondering if anyone has code to following: 1) Apply labels from a previous text classification dataset like this type of data (https://colab.research.google.com/drive/...
0
votes
1answer
58 views

text preprocessing using keras [closed]

I am getting started with NLP, in kaggle , and it dont get how this keras preprocessing works if anyone could explain the code would be much helpful,thanks ...
2
votes
0answers
232 views

Clustering as a method to find and label classes for supervised learning

I'm working on a text classification project. We have around 300k documents (small, 1~2 phrases) and we don't know the set of labels or how many labels there are. The recommended approach to me was ...
1
vote
0answers
353 views

Can cosine similarity be used to measure similarity between words?

In text mining books, I generally see cosine similarity used as a way to assess the similarity in documents; however, by transposing a tf-idf matrix, one can also calculate cosine similarity between ...
3
votes
1answer
381 views

Why does Naive Bayes work better when the number of features >> sample size compared to more sophisticated ML algorithms?

According to this article Because of the class independence assumption, naive Bayes classifiers can quickly learn to use high dimensional features with limited training data compared to more ...
1
vote
0answers
29 views

What is an appropriate Evaluation Metric and corresponding Loss function which best optimize the metric for a classification based FAQ Chatbot?

I am developing a FAQ chatbot to display/return only one correct answer in a chat window for a given question from the user. I know MRR & MAP make sense as an evaluation metric for information ...
0
votes
1answer
95 views

Applying PCA - First two components explain low variance but have high data separation when plotting

Applying PCA on a set of documents gives strange results in terms of the variance explained by the PCs vs the data separation I'm having when plotting the first two principle components. Details: ...
1
vote
1answer
100 views

How to implement topic modelling in regression analysis

I have a dataset consisting of hotel reviews, ratings, and other features such as traveller type, and word count of the review. I want to perform topic modeling (LDA) and use the topics derived from ...
0
votes
1answer
75 views

Text Similarity - Cosine - Control. Suggestion to another / better method?

I would like to ask you, if anybody could check my code, because it was behaving weird - not working, giving me errors to suddenly working without changing anything - the code will be at the bottom. ...
0
votes
0answers
21 views

Unsupervised learning with DNN on text

I want to: Have a key:pair database with author:largetextfileofeverysentencetheauthorpublished.txt Set up a deep neural network to see without supervision patterns in choice of vocabulary. Have the ...
0
votes
1answer
49 views

Comparison of text data for same distribution

I have two datasets with different text in it. I want to check if they are from the same distribution. If they were numbers tabulated in some way, it would be easy. Since this is text, how can this be ...
0
votes
0answers
131 views

Current SOTA in text classification

I've been recently starting with text classification, yet the amount of work in this field is somewhat overwhelming. Could you please direct me towards some of the SOTA (deep) approaches for solving ...
0
votes
0answers
15 views

Enrich true signal from loads of data

Background For simplicity let's say we have an alphabet of ABC and we are looking at words that all have the same length (n = 10)...
1
vote
0answers
39 views

HDP: Gibbs sampler implementation

I am trying to recreate the model proposed by Gao et al. (2011), based on the Hierarchical Dirichlet Process proposed by Teh and al. (2005). To estimate the model (let's call it iHDP) I need to ...