Questions tagged [text-mining]

Refers to a subset of data mining concerned with extracting information from data in the form of text by recognizing patterns. The goal of text mining is often to classify a given document into one of a number of categories in an automatic way, and to improve this performance dynamically, making it an example of machine learning. One example of this type of text mining are spam filters used for email.

Filter by
Sorted by
Tagged with
0
votes
0answers
8 views

document subsimilarity matching

I'm looking to classify subsections of "full" documents based on their similarity to a set of subsections that have been manually curated and assigned labels (let's call these short ...
0
votes
0answers
7 views

How do you use a TF-IDF matrix to score text similarity?

I'm trying to match strings of words with a website which has bulletpoints from all of the URL's I'm interested in whose text is most similar to it. The way I thought of doing it is to get all of the ...
1
vote
0answers
7 views

How would I create a machine learning based text parser? [closed]

I have many documents that have the same sections, but have different formatting. How could I make a model to segment the document into given sections? I have tried making classifiers for specific ...
0
votes
0answers
24 views

Machine Learning Method to Predict Continuous Target Variable with Text Data Using R

Purpose I have a surgery dataset with surgery id (i.e.,oid), the procedure names (o3.name.procedure), and the actual duration (i....
0
votes
0answers
9 views

Can we find the semantic similarity between two words in same document in encoded matrix from Latent Semantic Analysis?

Question The following the is encoded matrix obtained through Latent Semantic Analysis, my final goal is to find if we can get the semantic similarity between two words in one topic let's say topic 1 ...
1
vote
0answers
37 views

Alternative to tf-idf for 2 documents?

I am currently trying to construct word clouds between two tidy text documents (tweets). My question is methodological, although I am using bind_tf_idf in R for the ...
0
votes
0answers
43 views

What is a good method to compare the values with different sample sizes?

I am performing text analysis on some social media posts and I have a list of words and corresponding engagement rates, such as the list below (a much shorter version of a long list). As you can see, ...
0
votes
0answers
15 views

eBay search phrase to listings relevance

Working on a quick PoC, and it’s based on eBay data. Basically, someone puts in a search phrase and receives a number of listings. These listings have been classified (not by eBay, but by my colleague)...
3
votes
1answer
43 views

How are sentences one-hot encoded internally in an Embedding Keras Layer?

Multiple references are clear on how a single word is one-hot encoded in an Embedding layer, but what about sentences? In order to illustrate an example, I will use the following SO reference. Let's ...
0
votes
0answers
12 views

Can I use the masi distance to compare coders where the text document is not exactly the same?

It seems that Passoneau's MASI distance https://academiccommons.columbia.edu/doi/10.7916/D8PV6TTT was meant to grade different judges on the same document. Could I use this distance method to compare ...
0
votes
0answers
9 views

Should bag of words in training set include test set data when doing text classification?

I'm doing text classification to identify 'attacks' from wikipedia comments using a simple bag of words model and a linear SVM classifier. Because of class imbalance I'm using F1 score as my error ...
2
votes
0answers
29 views

Question related to using Pooled Output from BERT for similarity between sentences

I was hoping someone could give me advice and feedback on my current approach and possibly suggest to me a possible alternative. I'm trying to find the sentences that are most similar using the pooled ...
1
vote
0answers
83 views

How does the pooled output from the output layer in a BERT model reference back to the actual text?

I was wondering if someone can refer to me a source or describe to me how to interpret the 768 sequence of numbers that are derived from the output layer of the BERT Model. Like, what do they mean and ...
0
votes
0answers
18 views

How to perform text classification on unlabelled test data?

I am using TF-IDF to perform feature extraction and then passing the sparse matrix to perform training along with text data which is also transformed to sparse matrix. I understand that the input to ...
1
vote
1answer
124 views

Latent Dirichlet Allocation and topic distributions

When reading about the LDA, the generative procedure for a document is often presented as follows: For each topic $k\in \{1,\ldots, K\}$ Draw a distribution over words $\phi_k\sim \text{Dirichlet}(\...
0
votes
0answers
20 views

Sample size for doing topic modeling using LDA() in R (topicmodels package)

I just started to learn and do text analysis for open-text survey questions. My sample size is around 2000. I want to use the LDA() function in R (topicmodels package) to identify topics among the ...
1
vote
1answer
27 views

Best practice for named entity recognition on large texts

What are the best practices to apply NER to large texts (e.g 20 pages+)? One common advice is to split the text before passing it as input to the model. However this can require a significant manual ...
0
votes
0answers
13 views

How should data be organized for topic modelling

Say I have 10 documents with 10 sentences each. I'm curious how should my raw data look like? Is there a standard? Should it be like? ...
0
votes
0answers
25 views

Using Topic Modeling for Text Classification

I'm trying to use topic modeling using Latent Dirichlet Allocation as input for text classification problem. Although, I'm not getting good results by doing this. The data has three variables doc id, ...
0
votes
0answers
13 views

Suppressing false alarms with capturing information from unstructured corpus

In our team, we had previously deployed a machine learning anomaly detection tool at a chemical plant. It has been observed in certain cases that ongoing manual operations/interventions at plant can ...
0
votes
0answers
25 views

Information Retrieval and Event Prediction from Unstructured Document Corpus

My question is quite open ended. In some chemical plant, by using the sensor data available we first deployed a machine learning tool that can predict the onset on anomalous behavior with some decent ...
0
votes
0answers
12 views

Is it possible to use a dependent variable derived from one regression as an independent variable in a different regression?

For my thesis, I'm researching the effect of employee job satisfaction on customer satisfaction, mediated by service quality, using a company as a case study. I was planning to determine employee ...
0
votes
0answers
13 views

Difference between Structural Topic Modeling(STM) and SAGE (Sparse Additive Generative Model)?

I have read that STM combines 3 models of: (1) correlated topic model (CTM) (2) Dirichlet-Multinomial Regression (DMR) topic model (3) Sparse Additive Generative Model (SAGE) Is it correct to just ...
1
vote
1answer
86 views

Clustering after t-SNE in R

As explained here, t-SNE maps high dimensional data such as word embedding into a lower dimension in such that the distance between two words roughly describe the similarity. It also begins to create ...
1
vote
1answer
18 views

How many emails would I need to train a good text extraction model?

I'm looking to train a model that will identify product names in an email that a user has bought. The end result would be something very much like named entity extraction, except this should correctly ...
0
votes
0answers
7 views

Textmining classification: Using odds to generate a composite score

I have a classification task where i am predicting if someone owns a cat or not (made up example). I have columns which contain answers to questions in free form. For example "I own a feline cat&...
0
votes
0answers
12 views

How to perform large scale multihierarchical text classification?

There is a dataset from an old kaggle competition, https://www.kaggle.com/c/lshtc/discussion/7980 and I wanted to work on it as I am learning NLP. I have done a ...
0
votes
0answers
12 views

Estimating the best length of window for winnowing

I have to analyze biographies or descriptions for social media profiles (20-40 words) and compare them to the user input to check if we have found a correct person. What window length is it better to ...
0
votes
0answers
37 views

Determine if resume meets requirements of job description

What would be the best approach to determining if, or how much a resume meets the requirements of a job description. I understand you could extract features from both texts with Latent Dirichlet ...
0
votes
0answers
54 views

Should we remove duplicates in the context of text mining?

When we use clustering algorithms, removing duplicates might impact the results. For instance, k means might find different centroids. However, in the context of text analysis, we may have a really ...
0
votes
0answers
19 views

R or PIP Packages for Trend Analysis of String Data?

I have a bunch of strings coming in every day -- e.g. first names of people. I need to do some trend analysis or time series analysis to find the frequency of occurrence of each one, and to alert me ...
0
votes
0answers
11 views

Extrapolation of COVID cases based on textual analysis (ML)

I am just learning about machine learning and have strong interests in learning about how textual analysis from machine learning can be applied to time series prediction. A few examples I thought of ...
0
votes
0answers
19 views

Clustering mixed data based on text anlysis: Approach evaluation

As part of my project, I've been trying to analyse (and hopefully make some knowledgeable conclusions about) the movie database dataset, which consists of the following columns: Movie ID - ID of a ...
0
votes
0answers
19 views

Clustering mixed data based on text anlysis: Sparse Matrix problem

Good day/evening/any other time of the day! As part of my project, I've been trying to analyse (and hopefully make some konwlegable conclusions about) the movie database dataset, which consists of the ...
1
vote
0answers
36 views

When would you use purity as a measure of external validity over entropy? [closed]

This question particularly pertains to text clustering. I've not really found anything on why one would use purity over entropy or vice versa. Could someone explain this to me?
0
votes
0answers
32 views

When to perform feature selection?

I'm making my undergraduate thesis that proposed K-Nearest Neighbor and Chi-Square feature selection to do sentiment analysis. I also using TF-IDF as term weighting. My question: is feature selection ...
4
votes
1answer
201 views

How to use LDA to classify documents into pre defined topics

LDA is unsupervised and it classifies documents into topics. But, is there a way to make the LDA classify the documents into the predefined (or specific desired) topics. Below link says we need custom ...
0
votes
0answers
14 views

Best approach for clustering customer support requests (sentence form)?

I have a million records of customer support requests in sentence form. Something like this: ...
0
votes
0answers
54 views

What is the advantage of using lasso logistic regression in text classification cases?

I have tried to classify text using lasso logistic regression and it has a quite good f1 score, around 95%. But I have no specific reason at first time trying the methods. What is exactly the ...
1
vote
0answers
15 views

Test if semantic triple occurs more often than by chance

I have a large table (more than 9,000,000 rows) of semantic predications (i.e., triples of subject-predicate-object) extracted from sentences of scientific abstracts. The data are organized in the ...
0
votes
0answers
10 views

Applying cross-validation to find the best length of n-gram [duplicate]

Having seen questions on stackoverflow and stats.stackexchange, I have not found a hands-on example of using cross-validation for finding the best length of n-gram. I am writing a plagiarism detection ...
0
votes
0answers
20 views

word distribution similarity in different dataset

I have two datasets (A and B) with almost overlapped words. My goal is to check if the top k specific sequence of words has the same distribution in both datasets. What I did is as follows:\ I used ...
0
votes
0answers
12 views

How to compare 2 datasets and validate if there are similar rows

In the beginning this could be a simple query, but let me explain further. I have dataset A from company A and dataset B from company B. The datasets are about client data. They want to know the ...
1
vote
0answers
16 views

Question text Mining using Random Forest and PCA [closed]

I'm currently using the Reuters 50-50 dataset (https://archive.ics.uci.edu/ml/datasets/Reuter_50_50) to predict authorship. I've tried to first use PCA on both the test and training dataset to get a ...
0
votes
0answers
8 views

Text Analysis classifyihng authorship of documents

I'm trying to create a model that is able to predict authorship. I have multiple different documents from the various authors. I was hoping to get the TF IDF of all words from each of the documents ...
0
votes
1answer
17 views

Extracting information from form document through supervised learning

I was searching for a while around the web and I couldn't find any solution that would give some ideas on how to solve my problem. I have a few hundreds of document with some permission forms filled ...
0
votes
0answers
30 views

NLP: how to quantify information-richeness of short text (i.e., tweet)

I'm not very familliar with NLP or text mining, so forgive me if this is naive. Background I'm working on a personal project, where I fetch tweets from many people and then I try to do some filtering ...
1
vote
0answers
13 views

Multilabel Tweet Classification

I need some general advice and possible ideas. Problem statement goes like this -- We are given a tweet and we have to specify associated labels for it like generalized hate, support, oppose, ...
1
vote
0answers
23 views

Significance of datapoint outside prediction band

With linear regression I am plotting 25 bodies of text with their vocabulary count (independent variable X) and occurrence of a particular word (for example: "this"). I have a linear ...
0
votes
1answer
36 views

Feature Selection in Twitter Sentiment Analysis

I'm currently working on a twitter sentiment analysis project. In this project, one requirement is to perform Feature selection for a better prediction. But I'm fairly confused about the techniques to ...

1
2 3 4 5
14