Questions tagged [text-mining]

Refers to a subset of data mining concerned with extracting information from data in the form of text by recognizing patterns. The goal of text mining is often to classify a given document into one of a number of categories in an automatic way, and to improve this performance dynamically, making it an example of machine learning. One example of this type of text mining are spam filters used for email.

Filter by
Sorted by
Tagged with
0
votes
0answers
7 views

multi class text classification with multiple dependent variable to one set of predictors in r

I am doing multi class classification in text in r on a dataset containing two columns; feedback and topics. Some feedback has been assigned to more than one topics and some more than two but most of ...
0
votes
0answers
14 views

Text Match for Description Fields

I have a long file with the list of item id's and their descriptions. These descriptions are like "The Brawn White Bolt Laser 10W" and "Laser for 10 Watt White Bolt" with different item id's. Though ...
-1
votes
1answer
16 views

Clustering text file into segments [closed]

I have a big text file (over 5 GB) of log files from some network devices. The log consists of outputs from these devices after performing many different commands on them. However outputs are not ...
0
votes
0answers
21 views

Why is my correlation matrix dropping that many NA?

I am trying to build a correlation matrix among documents per topic on a Latent Dirichlet Allocation model by text2vec, getting a doc_topic_distr matrix like below, with only first 5 documents, it's a ...
0
votes
0answers
6 views

Overlapping meaning with lda_model$plot()

I'm doing Latent Dirichlet Allocation with text2vec and end up with lda_model$plot() visualization from LDAvis package but I don't specifically understand what is the overlapping meaning with it. I ...
31
votes
2answers
653 views

StackExchange fires a moderator, and now in response hundreds of moderators resign: is the increase in resignations statistically significant?

I am doing a study on StackExchange. The management of StackExchange has demodded (for unclear reasons) a moderator, and now the network is on fire. Currently many moderators resign or suspend their ...
0
votes
0answers
13 views

Does it make sense to apply Latent Dirichlet Allocantion on topic outcomes from the model?

What I mean is that if it makes sense to apply LDA twice, the first time as it is supossed to and the second apllying the model to each topic outcome from the first application? If it is possible, ...
1
vote
1answer
31 views

Finding a varying code into a text

I'm rather new to Machine Learning but I have been looking into it for a bit now. Specially I've been interested in text classifying solutions and seen how a high level of success has been achieved in ...
0
votes
0answers
10 views

Difference lemmatizing/stemming when preprocessing text to organize abstracts looking for document insights?

I'm working with R text2vec in order to apply an LDA on a 230k text data that I have on hand. I tried both stemming and lemmatizing separately but I am not completely aware of which gives out the ...
2
votes
0answers
16 views

Beginner question: abstracts comparison

We want to compare the abstract of a chosen article with that of several article, to identify those that would be more "relevant" from a content point of view. Question: Do you know any tool/method ...
1
vote
0answers
8 views

Recommendation Engine and Text Analytics

I am looking for a dataset on which I can use Collaborative filtering and Content based filtering along with Text Mining. Could anybody please suggest , is there any dataset on which I can apply ...
0
votes
0answers
38 views

Estimating Required Data for NLP Classification Models

Are there general guidelines for how much data is required for natural language processing (NLP) classification models? I understand this may depend on the text quality, text length, how accurate the ...
2
votes
0answers
20 views

Variable selection, variable reduction, and handling sparsity for binary text classification

I am trying to do a binary text classification using support vector machine. I am wondering if I am doing it right and I'd like to look for some answers to the questions in mind. The following ...
1
vote
2answers
112 views

Text classification with small dataset for a specialized domain

I have a multiclass text classification problem where I have very few documents for each class. The classes are imbalanced but I want to be able to predict the class when I have at least 200 - 300 ...
0
votes
0answers
32 views

What does a high cosine similarity score mean here?

I have a set of 50,000 documents divided into two classes. Class 1 has 5000 documents and, Class 2 has 45,000 documents Using word2vec embeddings, I extract 300 dimensional dense, real-valued ...
0
votes
1answer
39 views

Classification Model - How to Preprocess Text

I have a Dataframe that contains 2 columns: 'Skills' column - each cell contains a list of strings describing different technical and soft skills of a person, e.g: [Python,SQL,Java,Team Management,...
0
votes
0answers
29 views

Hierarchical Text classification

I am working on a project with a huge number of big groups of sentences! I need to classify each sentence based on the sentence and other sentences on the group. Actually there is 2 level of ...
0
votes
1answer
29 views

Text classification suggestion

I have a big data-set of sentences (tens of thousands) which some of these sentences are big but some are short. The main problem is that you should classify some sentences according to previous or ...
0
votes
0answers
15 views

How do I use additional features when doing sentiment analysis?

Say I have a dataset of 3 columns: text, topic and sentiment. Each row of the ...
0
votes
1answer
262 views

Hoes does laplace smoothing in Naive Bayes control high bias and high variance?

I'm trying to understand how laplace smoothing exactly helps to balance between overfitting and underfitting. I know that Laplace smoothing is used as a fail safe probability if there's a any ...
0
votes
0answers
162 views

TF-IDF versus TF for Cosine Similarity

My task is to examine emails and determine the cosine similarity between pairs of emails to see which ones are the same or almost the same. I was thinking of using the TF-IDF technique, but am not ...
0
votes
0answers
24 views

binary classification problem for data where one class potentially includes members of the other class

Situation Assuming one wants to classify text data into two sentiment categories: negative and neutral (i.e everything that is not negative). For the training process, labelled data is available. ...
0
votes
0answers
23 views

Classify documents using a set of known vocabularies

I have a bunch of documents that I want to classify which ones talk about soccer (unsupervised learning, I do not want to manually label the documents). One way I am thinking about is to go online ...
0
votes
0answers
30 views

Improve the accuracy of semantic text matching

I have a corpus of ~200K sentences of variable length, the median length is 16 words. My goal is for a given sentence to find other sentences with a similar meaning. I tried several approaches: using ...
0
votes
1answer
19 views

Should I convert classification output to integer and how?

I'm using a neural network to classify text, and the label of the training data is 0 or 1(i.e. binary classification). It works well in the training and evaluating process, but the prediction output ...
0
votes
0answers
14 views

classification association rules and closed frequent itemsets

if I need to mine all frequent itemsets for text classification using classification association rules. what is the most efficient algorithm? is CHARM algorithm efficient with mining only closed ...
0
votes
0answers
64 views

Visualizing a Latent Dirichlet Allocation (LDA) by Multidimensional Scaling (MDS)

I did an LDA with four topics for four different Smartphones. This was done using customer Reviews of Amazon. ...
0
votes
0answers
5 views

Combine Noisy predictions from an optical character recognition program

I am trying to perform optical character recognition on a field of text from different angles as the camera passes over it and beyond it. Due to the 3D skewing of the image, thee readings of the OCR ...
0
votes
1answer
27 views

Get association of categorical variables

I do a text analysis where I want to identify dependencies among categorical variables, for example let's take this dataset: ...
0
votes
0answers
127 views

Using LDA to create two layered wordcloud

I create a word-cloud using LDA model what I want to do is to find the documents IDs related to that topic group. So, for example, the image here I want to allow users to click the word broccoli and ...
0
votes
0answers
15 views

Correlations in text analysis - for an absolute beginner

I am analysing the media releases and speeches of two political leaders. I want to find out: how often they mention young people, and in what context. I know my frequency tables are correct because I ...
0
votes
0answers
55 views

How PV-DBOW works

The authors of the Paragraph Vector paper describe PV-DBOW with: 2.3. Paragraph Vector without word ordering: Distributed bag of words The above method considers the concatenation of the ...
0
votes
1answer
214 views

How to find correlation between text data?

I have data set similar to this: I want to if the columns subtype and item are correlated. They have different text, ...
0
votes
0answers
14 views

How to solve the estimate population characters with sample text-mining?

I've recently learning text-mining, but none of my textbooks talk of inferential statistics; they talk about how to analyze collected data but hardly deal with how to estimate population data through ...
0
votes
0answers
8 views

wordfish - must reference texts be at the absolute extreme of the spectrum

I have a question pertaining to the wordfish method (see here for the academic paper introducing the package and here for the package itself). My question regards ...
0
votes
0answers
23 views

How to statistically infer common pattern in text

Am trying to solve a problem where I need to infer common patterns in text for example, the data below, with bare eyes it can be noticed there is a pattern and that is ...
0
votes
0answers
12 views

Calculating the ratio of the (frequency of a specific word in the corpus/sum of the frequency of all the words)

I have the following code which gives me the list of top 10 words in a corpus in descending order of frequency: library(tidytext) tidybooks.nstop<-tidy_books %>% + anti_join(stop_words) ...
0
votes
0answers
25 views

Classification of sample with only unseen words

I'm doing text classification (Product Name) where one example belongs to one class. "Some Product Name" -> MODEL -> {CLASS_1 | CLASS_2 | CLASS_3 | CLASS_4} ...
2
votes
0answers
118 views

How to compute gain statistic for the multinomial Naive Bayes classifier from Jurafsky and Martin (2018)

I'm trying to figure out how to compute the gain statistic G(w) following the fitting of the multinomial Naive Bayes model. This statistic is described on p17 of the new edition of Jurafsky and ...
0
votes
0answers
36 views

Sentiment Analysis Issues Using R

I am using the R package sentimentr and sentiment function to get polarity/sentiment scores on a list of comments. The issue I am having is that the comments are in ...
0
votes
0answers
51 views

Difference between Semantic analysis and Syntactic analysis in text analysis

Can someone explain me clearly the meaning of Semantic analysis and Syntactic Analysis in text analysis ? I am very much confused between these two. Also tell the difference in how they are used in ...
0
votes
0answers
28 views

How to ensemble predictions from image classifier and text classifier?

I am doing multiclass classification based on images and text. I have predictions from both image classification and text. I am not sure how to combine them. Should I use probabilities as a feature to ...
0
votes
0answers
61 views

Guided LDA to categorize software requirements

I'm developing a application to categorize requirements in a requirement specification in to categories like database, front end, back end, etc. So for that I'm trying to use Guided LDA since labelled ...
0
votes
0answers
35 views

Pointwise Mutual Information Word Dependency

I have pre-defined concepts which are either a single word or couple of words that refer to a concept.( In the context of machine learning for instance, covariance matrix is a concept). I am trying to ...
2
votes
0answers
15 views

Correlation of phrases and answers to them

If I ask question in wrong forum, let me know, I'll delete it. I try find methods, methodology to create predicative model. Namely, I want to investigate the relationship between the sentences and ...
0
votes
0answers
18 views

naive bayes text classification: can I look at the individual word probabilities?

Assume I use the Naive Bayes classification algorithm. My question is simple: can I rank the words according to their posterior probabilities? I want to have a measure of "importance" of the words in ...
1
vote
1answer
17 views

Good way to use word similarity as a feature in supervised ML on text

I have a pretty low N data set of small sentences tagged with a label. I would like to create a classifier on this dataset. The word choice is not very variable since the domain is pretty specific. ...
0
votes
0answers
40 views

Using a priori knowledge in a classification task

I'm working on a classification task, related with text classification, where texts to be classified are requests for technical support, and the classes are technical guys which issues can be assigned ...
1
vote
1answer
362 views

Finding similar text - algorithms and evaluation

I've been asked to create a program that will rank similar texts to an input text given a collection of text. So far I've been using a tdidf representation and cosine similarity with a lot of regex-...
0
votes
0answers
76 views

Getting the document-per-topic loading using TextmineR package by passing term co-occurrence matrix

I am using TextmineR package to find the most similar documents to given document list. I used the following code to generate the tcm not dtm ...