Questions tagged [text-mining]

Refers to a subset of data mining concerned with extracting information from data in the form of text by recognizing patterns. The goal of text mining is often to classify a given document into one of a number of categories in an automatic way, and to improve this performance dynamically, making it an example of machine learning. One example of this type of text mining are spam filters used for email.

Filter by
Sorted by
Tagged with
0
votes
0answers
4 views

Classification of Text (Comments from a Survey) in R using Latent Dirichlet Allocation

I have a couple of questions on how to classify a column of columns into distinct topics: How do you categorize the comments into k distinct topics that are important topics? How do you choose the ...
0
votes
0answers
24 views
+50

A little help on text classification

Right now, I am working on building a efficient classification for my company. We work as a social monitoring company, basically we collect data from social media sites to see the engagement, comment, ...
0
votes
1answer
20 views

Text composition based on categorical features

The problem I have to solve is to find a model that links categorical features (bool type actually) to text documents. The categorical features are answers to questions. Any different combination of ...
0
votes
0answers
12 views

Dealing with an imbalanced dataset in text mining

As an English major with no traditional training in statistics, I am having a very rough time with this, so any help would be greatly appreciated. My problem is that only 849 books out of my 6360 book ...
0
votes
1answer
22 views

Combining two sequences for text classification

I'm doing text classification on comments posted on articles/stories. The two human-labeled classes are appropriate and not appropriate (not the same as happy/angry or any "sentiment" ...
0
votes
1answer
16 views

Are the vectorization settings considered hyperparameters in ML?

Short definition of HP: "In machine learning, a hyperparameter is a parameter whose value is set before the learning process begins. Hyperparameter optimization or tuning is the problem of ...
1
vote
1answer
16 views

Using POS Tags and NERs as Features for Text Classification or Sentiment Analysis

I am trying to implement text classification and sentiment analysis from the documents. I always use POS tags as features in the following way. Mike is playing football I would convert it into ...
1
vote
0answers
11 views

Multi-label Text class

The data i am dealing with are simple text sentences that needs to be classified into variaous labels that correspond to the different topics as simple as Yes/No class. Several labels can be assigned ...
0
votes
0answers
15 views

Text Analysis: creating a word cloud and how to get the most from text data

I want to create a word cloud from a data set. The data is a number of comments from people around the struggle they are having being unable to leave the home as a result of the corona virus. The ...
0
votes
0answers
6 views

Where the embeddings should be implemented in the RNN model?

Hi All (it's my first question here so welcome everyone), I wrote simple RNN model in tensor flow and I cannot figure out where the embeddings should be inserted inside, please find my code and below ...
0
votes
0answers
28 views

Clustering documents of text sequences (not in plain English) using 1D CNN without pre-trained word embedding

I have a long sequence of hex (or integer) numbers, each of which corresponding to an event. There are thousands of events per document, and I have several hundreds of documents. I’d like to do the ...
0
votes
0answers
8 views

why k-means is better in clustering than topic modelling algorithms like LDA?

I want to know about the advantages of K-means in clustering essays to discover their topics. There are a lot of algorithms to do it such as K-medoid, x-means, LDA, LSA, etc. Please give me a full ...
0
votes
0answers
20 views

Real life class imbalance [duplicate]

Fellow like-minded people, I'm writing my thesis in fake news detection on scrapped twitter data and facing an issue (among many others). Fake news consist of less than 10% of the total tweets or ...
0
votes
0answers
9 views

Text classification on a small unbalanced dataset: using externally derived features

Text classification on a small unbalanced dataset of text documents (N=479; label 1: N=404 , label 2: N=44 label 3: N=31) the 3rd label contains conspiratorial documents. Since I have so few examples ...
0
votes
0answers
10 views

Doc2Vec score keep getting worse

I'm using Doc2Vec on kaggle with XGB and MLPClassifier but i noticed that for five times in a row the roc scorse got worse without me changing the code (from 90 to 87). I set a fixed random state for ...
1
vote
1answer
107 views

ROUGE scores for extractive vs abstractive text summarization

The ROUGE score (scores) allows us to measure (although not in a perfect way) the quality of our text summarization by computing the frequency of overlapping n-grams between our produced summary and ...
0
votes
0answers
34 views

Hierarchical SoftMax for Skip Gram?

While I am reading the following article on the Internet, I am kind of feeling that I am not getting the full understanding of the picture. https://d2l.ai/chapter_natural-language-processing/approx-...
0
votes
0answers
10 views

Extract core business keywords from very short text

Recently I had the chance to get some publicly available data about companies. Data refers to company website and they are made up of keyword for SEO ( i suppose) and a brief description of the ...
0
votes
0answers
27 views

multi class text classification with multiple dependent variable to one set of predictors in r

I am doing multi class classification in text in r on a dataset containing two columns; feedback and topics. Some feedback has been assigned to more than one topics and some more than two but most of ...
0
votes
0answers
23 views

Text Match for Description Fields

I have a long file with the list of item id's and their descriptions. These descriptions are like "The Brawn White Bolt Laser 10W" and "Laser for 10 Watt White Bolt" with different item id's. Though ...
-1
votes
1answer
17 views

Clustering text file into segments [closed]

I have a big text file (over 5 GB) of log files from some network devices. The log consists of outputs from these devices after performing many different commands on them. However outputs are not ...
0
votes
0answers
23 views

Why is my correlation matrix dropping that many NA?

I am trying to build a correlation matrix among documents per topic on a Latent Dirichlet Allocation model by text2vec, getting a doc_topic_distr matrix like below, with only first 5 documents, it's a ...
0
votes
0answers
12 views

Overlapping meaning with lda_model$plot()

I'm doing Latent Dirichlet Allocation with text2vec and end up with lda_model$plot() visualization from LDAvis package but I don't specifically understand what is the overlapping meaning with it. I ...
31
votes
2answers
672 views

StackExchange fires a moderator, and now in response hundreds of moderators resign: is the increase in resignations statistically significant?

I am doing a study on StackExchange. The management of StackExchange has demodded (for unclear reasons) a moderator, and now the network is on fire. Currently many moderators resign or suspend their ...
0
votes
0answers
13 views

Does it make sense to apply Latent Dirichlet Allocantion on topic outcomes from the model?

What I mean is that if it makes sense to apply LDA twice, the first time as it is supossed to and the second apllying the model to each topic outcome from the first application? If it is possible, ...
1
vote
1answer
31 views

Finding a varying code into a text

I'm rather new to Machine Learning but I have been looking into it for a bit now. Specially I've been interested in text classifying solutions and seen how a high level of success has been achieved in ...
0
votes
0answers
11 views

Difference lemmatizing/stemming when preprocessing text to organize abstracts looking for document insights?

I'm working with R text2vec in order to apply an LDA on a 230k text data that I have on hand. I tried both stemming and lemmatizing separately but I am not completely aware of which gives out the ...
2
votes
0answers
16 views

Beginner question: abstracts comparison

We want to compare the abstract of a chosen article with that of several article, to identify those that would be more "relevant" from a content point of view. Question: Do you know any tool/method ...
1
vote
0answers
9 views

Recommendation Engine and Text Analytics

I am looking for a dataset on which I can use Collaborative filtering and Content based filtering along with Text Mining. Could anybody please suggest , is there any dataset on which I can apply ...
0
votes
0answers
41 views

Estimating Required Data for NLP Classification Models

Are there general guidelines for how much data is required for natural language processing (NLP) classification models? I understand this may depend on the text quality, text length, how accurate the ...
2
votes
0answers
22 views

Variable selection, variable reduction, and handling sparsity for binary text classification

I am trying to do a binary text classification using support vector machine. I am wondering if I am doing it right and I'd like to look for some answers to the questions in mind. The following ...
1
vote
2answers
262 views

Text classification with small dataset for a specialized domain

I have a multiclass text classification problem where I have very few documents for each class. The classes are imbalanced but I want to be able to predict the class when I have at least 200 - 300 ...
0
votes
0answers
36 views

What does a high cosine similarity score mean here?

I have a set of 50,000 documents divided into two classes. Class 1 has 5000 documents and, Class 2 has 45,000 documents Using word2vec embeddings, I extract 300 dimensional dense, real-valued ...
0
votes
1answer
41 views

Classification Model - How to Preprocess Text

I have a Dataframe that contains 2 columns: 'Skills' column - each cell contains a list of strings describing different technical and soft skills of a person, e.g: [Python,SQL,Java,Team Management,...
0
votes
0answers
33 views

Hierarchical Text classification

I am working on a project with a huge number of big groups of sentences! I need to classify each sentence based on the sentence and other sentences on the group. Actually there is 2 level of ...
0
votes
1answer
40 views

Text classification suggestion

I have a big data-set of sentences (tens of thousands) which some of these sentences are big but some are short. The main problem is that you should classify some sentences according to previous or ...
0
votes
0answers
16 views

How do I use additional features when doing sentiment analysis?

Say I have a dataset of 3 columns: text, topic and sentiment. Each row of the ...
0
votes
1answer
506 views

Hoes does laplace smoothing in Naive Bayes control high bias and high variance?

I'm trying to understand how laplace smoothing exactly helps to balance between overfitting and underfitting. I know that Laplace smoothing is used as a fail safe probability if there's a any ...
0
votes
1answer
26 views

Should I convert classification output to integer and how?

I'm using a neural network to classify text, and the label of the training data is 0 or 1(i.e. binary classification). It works well in the training and evaluating process, but the prediction output ...
0
votes
1answer
37 views

Get association of categorical variables

I do a text analysis where I want to identify dependencies among categorical variables, for example let's take this dataset: ...
0
votes
1answer
397 views

How to find correlation between text data?

I have data set similar to this: I want to if the columns subtype and item are correlated. They have different text, ...
2
votes
0answers
144 views

How to compute gain statistic for the multinomial Naive Bayes classifier from Jurafsky and Martin (2018)

I'm trying to figure out how to compute the gain statistic G(w) following the fitting of the multinomial Naive Bayes model. This statistic is described on p17 of the new edition of Jurafsky and ...
2
votes
0answers
15 views

Correlation of phrases and answers to them

If I ask question in wrong forum, let me know, I'll delete it. I try find methods, methodology to create predicative model. Namely, I want to investigate the relationship between the sentences and ...
1
vote
1answer
17 views

Good way to use word similarity as a feature in supervised ML on text

I have a pretty low N data set of small sentences tagged with a label. I would like to create a classifier on this dataset. The word choice is not very variable since the domain is pretty specific. ...
0
votes
0answers
64 views

Using a priori knowledge in a classification task

I'm working on a classification task, related with text classification, where texts to be classified are requests for technical support, and the classes are technical guys which issues can be assigned ...
1
vote
1answer
498 views

Finding similar text - algorithms and evaluation

I've been asked to create a program that will rank similar texts to an input text given a collection of text. So far I've been using a tdidf representation and cosine similarity with a lot of regex-...
0
votes
1answer
74 views

apply CountVectorized to whole data before applying train_test_split

Is there any difference between the two different snips of codes. ...
2
votes
1answer
23 views

Information theoretic alternative to tf-idf heuristic?

I've been recently working with feature construction from texts, where tf-idf measure is one of the main options for vectorizing the documents (one feature per e.g., word). I was wondering, whether ...
2
votes
1answer
2k views

Imbalanced multiclass classification with many classes

I am working on a text classification project in which we have hundreds of (imbalanced) classes. Some characteristics of the data: We have examples of "bad" documents. Basically documents that don't ...
0
votes
1answer
129 views

text preprocessing using keras [closed]

I am getting started with NLP, in kaggle , and it dont get how this keras preprocessing works if anyone could explain the code would be much helpful,thanks ...

1
2 3 4 5
13