Questions tagged [text-mining]

Refers to a subset of data mining concerned with extracting information from data in the form of text by recognizing patterns. The goal of text mining is often to classify a given document into one of a number of categories in an automatic way, and to improve this performance dynamically, making it an example of machine learning. One example of this type of text mining are spam filters used for email.

Filter by
Sorted by
Tagged with
0
votes
0answers
4 views

Difference between Structural Topic Modeling(STM) and SAGE (Sparse Additive Generative Model)?

I have read that STM combines 3 models of: (1) correlated topic model (CTM) (2) Dirichlet-Multinomial Regression (DMR) topic model (3) Sparse Additive Generative Model (SAGE) Is it correct to just ...
1
vote
1answer
38 views

Clustering after t-SNE in R

As explained here, t-SNE maps high dimensional data such as word embedding into a lower dimension in such that the distance between two words roughly describe the similarity. It also begins to create ...
1
vote
1answer
15 views

How many emails would I need to train a good text extraction model?

I'm looking to train a model that will identify product names in an email that a user has bought. The end result would be something very much like named entity extraction, except this should correctly ...
0
votes
0answers
7 views

Textmining classification: Using odds to generate a composite score

I have a classification task where i am predicting if someone owns a cat or not (made up example). I have columns which contain answers to questions in free form. For example "I own a feline cat&...
0
votes
0answers
7 views

How to perform large scale multihierarchical text classification?

There is a dataset from an old kaggle competition, https://www.kaggle.com/c/lshtc/discussion/7980 and I wanted to work on it as I am learning NLP. I have done a ...
0
votes
0answers
11 views

Estimating the best length of window for winnowing

I have to analyze biographies or descriptions for social media profiles (20-40 words) and compare them to the user input to check if we have found a correct person. What window length is it better to ...
0
votes
0answers
37 views

Determine if resume meets requirements of job description

What would be the best approach to determining if, or how much a resume meets the requirements of a job description. I understand you could extract features from both texts with Latent Dirichlet ...
0
votes
0answers
20 views

Should we remove duplicates in the context of text mining?

When we use clustering algorithms, removing duplicates might impact the results. For instance, k means might find different centroids. However, in the context of text analysis, we may have a really ...
0
votes
0answers
19 views

R or PIP Packages for Trend Analysis of String Data?

I have a bunch of strings coming in every day -- e.g. first names of people. I need to do some trend analysis or time series analysis to find the frequency of occurrence of each one, and to alert me ...
0
votes
0answers
10 views

Extrapolation of COVID cases based on textual analysis (ML)

I am just learning about machine learning and have strong interests in learning about how textual analysis from machine learning can be applied to time series prediction. A few examples I thought of ...
0
votes
0answers
18 views

Clustering mixed data based on text anlysis: Approach evaluation

As part of my project, I've been trying to analyse (and hopefully make some knowledgeable conclusions about) the movie database dataset, which consists of the following columns: Movie ID - ID of a ...
0
votes
0answers
17 views

Clustering mixed data based on text anlysis: Sparse Matrix problem

Good day/evening/any other time of the day! As part of my project, I've been trying to analyse (and hopefully make some konwlegable conclusions about) the movie database dataset, which consists of the ...
1
vote
0answers
34 views

When would you use purity as a measure of external validity over entropy? [closed]

This question particularly pertains to text clustering. I've not really found anything on why one would use purity over entropy or vice versa. Could someone explain this to me?
0
votes
0answers
25 views

When to perform feature selection?

I'm making my undergraduate thesis that proposed K-Nearest Neighbor and Chi-Square feature selection to do sentiment analysis. I also using TF-IDF as term weighting. My question: is feature selection ...
3
votes
1answer
110 views

How to use LDA to classify documents into pre defined topics

LDA is unsupervised and it classifies documents into topics. But, is there a way to make the LDA classify the documents into the predefined (or specific desired) topics. Below link says we need custom ...
0
votes
0answers
13 views

Best approach for clustering customer support requests (sentence form)?

I have a million records of customer support requests in sentence form. Something like this: ...
0
votes
0answers
26 views

What is the advantage of using lasso logistic regression in text classification cases?

I have tried to classify text using lasso logistic regression and it has a quite good f1 score, around 95%. But I have no specific reason at first time trying the methods. What is exactly the ...
1
vote
0answers
12 views

Test if semantic triple occurs more often than by chance

I have a large table (more than 9,000,000 rows) of semantic predications (i.e., triples of subject-predicate-object) extracted from sentences of scientific abstracts. The data are organized in the ...
0
votes
0answers
9 views

Applying cross-validation to find the best length of n-gram [duplicate]

Having seen questions on stackoverflow and stats.stackexchange, I have not found a hands-on example of using cross-validation for finding the best length of n-gram. I am writing a plagiarism detection ...
0
votes
0answers
17 views

word distribution similarity in different dataset

I have two datasets (A and B) with almost overlapped words. My goal is to check if the top k specific sequence of words has the same distribution in both datasets. What I did is as follows:\ I used ...
0
votes
0answers
11 views

How to compare 2 datasets and validate if there are similar rows

In the beginning this could be a simple query, but let me explain further. I have dataset A from company A and dataset B from company B. The datasets are about client data. They want to know the ...
1
vote
0answers
15 views

Question text Mining using Random Forest and PCA [closed]

I'm currently using the Reuters 50-50 dataset (https://archive.ics.uci.edu/ml/datasets/Reuter_50_50) to predict authorship. I've tried to first use PCA on both the test and training dataset to get a ...
0
votes
0answers
8 views

Text Analysis classifyihng authorship of documents

I'm trying to create a model that is able to predict authorship. I have multiple different documents from the various authors. I was hoping to get the TF IDF of all words from each of the documents ...
0
votes
1answer
15 views

Extracting information from form document through supervised learning

I was searching for a while around the web and I couldn't find any solution that would give some ideas on how to solve my problem. I have a few hundreds of document with some permission forms filled ...
0
votes
0answers
23 views

NLP: how to quantify information-richeness of short text (i.e., tweet)

I'm not very familliar with NLP or text mining, so forgive me if this is naive. Background I'm working on a personal project, where I fetch tweets from many people and then I try to do some filtering ...
1
vote
0answers
12 views

Multilabel Tweet Classification

I need some general advice and possible ideas. Problem statement goes like this -- We are given a tweet and we have to specify associated labels for it like generalized hate, support, oppose, ...
1
vote
0answers
20 views

Significance of datapoint outside prediction band

With linear regression I am plotting 25 bodies of text with their vocabulary count (independent variable X) and occurrence of a particular word (for example: "this"). I have a linear ...
0
votes
1answer
25 views

Feature Selection in Twitter Sentiment Analysis

I'm currently working on a twitter sentiment analysis project. In this project, one requirement is to perform Feature selection for a better prediction. But I'm fairly confused about the techniques to ...
0
votes
1answer
32 views

Text composition based on categorical features

The problem I have to solve is to find a model that links categorical features (bool type actually) to text documents. The categorical features are answers to questions. Any different combination of ...
0
votes
0answers
14 views

Dealing with an imbalanced dataset in text mining

As an English major with no traditional training in statistics, I am having a very rough time with this, so any help would be greatly appreciated. My problem is that only 849 books out of my 6360 book ...
0
votes
1answer
39 views

Combining two sequences for text classification

I'm doing text classification on comments posted on articles/stories. The two human-labeled classes are appropriate and not appropriate (not the same as happy/angry or any "sentiment" ...
0
votes
1answer
25 views

Are the vectorization settings considered hyperparameters in ML?

Short definition of HP: "In machine learning, a hyperparameter is a parameter whose value is set before the learning process begins. Hyperparameter optimization or tuning is the problem of ...
1
vote
1answer
127 views

Using POS Tags and NERs as Features for Text Classification or Sentiment Analysis

I am trying to implement text classification and sentiment analysis from the documents. I always use POS tags as features in the following way. Mike is playing football I would convert it into ...
1
vote
0answers
14 views

Multi-label Text class

The data i am dealing with are simple text sentences that needs to be classified into variaous labels that correspond to the different topics as simple as Yes/No class. Several labels can be assigned ...
0
votes
0answers
16 views

Text Analysis: creating a word cloud and how to get the most from text data

I want to create a word cloud from a data set. The data is a number of comments from people around the struggle they are having being unable to leave the home as a result of the corona virus. The ...
0
votes
0answers
6 views

Where the embeddings should be implemented in the RNN model?

Hi All (it's my first question here so welcome everyone), I wrote simple RNN model in tensor flow and I cannot figure out where the embeddings should be inserted inside, please find my code and below ...
0
votes
0answers
37 views

Clustering documents of text sequences (not in plain English) using 1D CNN without pre-trained word embedding

I have a long sequence of hex (or integer) numbers, each of which corresponding to an event. There are thousands of events per document, and I have several hundreds of documents. I’d like to do the ...
0
votes
0answers
26 views

why k-means is better in clustering than topic modelling algorithms like LDA?

I want to know about the advantages of K-means in clustering essays to discover their topics. There are a lot of algorithms to do it such as K-medoid, x-means, LDA, LSA, etc. Please give me a full ...
0
votes
0answers
24 views

Real life class imbalance [duplicate]

Fellow like-minded people, I'm writing my thesis in fake news detection on scrapped twitter data and facing an issue (among many others). Fake news consist of less than 10% of the total tweets or ...
0
votes
0answers
12 views

Text classification on a small unbalanced dataset: using externally derived features

Text classification on a small unbalanced dataset of text documents (N=479; label 1: N=404 , label 2: N=44 label 3: N=31) the 3rd label contains conspiratorial documents. Since I have so few examples ...
0
votes
0answers
13 views

Transformation of SVD for latent semantic analysis

General Idea: I'm working through a particular implementation of Latent Semantic Analysis via SVD. Here is some example code. fairly simple: ...
0
votes
0answers
12 views

Doc2Vec score keep getting worse

I'm using Doc2Vec on kaggle with XGB and MLPClassifier but i noticed that for five times in a row the roc scorse got worse without me changing the code (from 90 to 87). I set a fixed random state for ...
1
vote
1answer
377 views

ROUGE scores for extractive vs abstractive text summarization

The ROUGE score (scores) allows us to measure (although not in a perfect way) the quality of our text summarization by computing the frequency of overlapping n-grams between our produced summary and ...
0
votes
0answers
83 views

Hierarchical SoftMax for Skip Gram?

While I am reading the following article on the Internet, I am kind of feeling that I am not getting the full understanding of the picture. https://d2l.ai/chapter_natural-language-processing/approx-...
0
votes
0answers
10 views

Extract core business keywords from very short text

Recently I had the chance to get some publicly available data about companies. Data refers to company website and they are made up of keyword for SEO ( i suppose) and a brief description of the ...
-1
votes
1answer
17 views

Clustering text file into segments [closed]

I have a big text file (over 5 GB) of log files from some network devices. The log consists of outputs from these devices after performing many different commands on them. However outputs are not ...
0
votes
0answers
33 views

Why is my correlation matrix dropping that many NA?

I am trying to build a correlation matrix among documents per topic on a Latent Dirichlet Allocation model by text2vec, getting a doc_topic_distr matrix like below, with only first 5 documents, it's a ...
32
votes
2answers
691 views

StackExchange fires a moderator, and now in response hundreds of moderators resign: is the increase in resignations statistically significant?

I am doing a study on StackExchange. The management of StackExchange has demodded (for unclear reasons) a moderator, and now the network is on fire. Currently many moderators resign or suspend their ...
1
vote
1answer
31 views

Finding a varying code into a text

I'm rather new to Machine Learning but I have been looking into it for a bit now. Specially I've been interested in text classifying solutions and seen how a high level of success has been achieved in ...
2
votes
0answers
21 views

Beginner question: abstracts comparison

We want to compare the abstract of a chosen article with that of several article, to identify those that would be more "relevant" from a content point of view. Question: Do you know any tool/method ...

1
2 3 4 5
13