Stack Exchange Network

Stack Exchange network consists of 174 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [text-mining]

Refers to a subset of data mining concerned with extracting information from data in the form of text by recognizing patterns. The goal of text mining is often to classify a given document into one of a number of categories in an automatic way, and to improve this performance dynamically, making it ...

0
votes
0answers
8 views

What is the best model for keyphrase extraction from super long text?

I’m working on a keyphrase extraction task. The biggest difficulty of this task is that the text is very long (5000-20000 words). I’ve tried several unsupervised algorithms such as Tf-idf and TextRank ...
0
votes
0answers
6 views

dataset for multi-document summarization [on hold]

I want to work on multi-document summarization task. DUC2007 is a multi-document summarization dataset. What other datasets are available in the context of multi-document summarization?
0
votes
1answer
29 views

Imbalanced multiclass classification with many classes

I am working on a text classification project in which we have hundreds of (imbalanced) classes. Some characteristics of the data: We have examples of "bad" documents. Basically documents that don't ...
0
votes
0answers
13 views

Applying Label From Supervised Learning to Unlabeled Data- Text Classification

I am wondering if anyone has code to following: 1) Apply labels from a previous text classification dataset like this type of data (https://colab.research.google.com/drive/...
0
votes
0answers
16 views

Find cars' trim levels from description [closed]

I have some descriptions of cars. For example, the description of Mercedes Benz (make) B 180 (model): "Mercedes-Benz B 180 B180 Sport" "Mercedes-Benz B 180 B180 Blue Efficiency S" "Mercedes-Benz B ...
0
votes
1answer
14 views

text preprocessing using keras [closed]

I am getting started with NLP, in kaggle , and it dont get how this keras preprocessing works if anyone could explain the code would be much helpful,thanks ...
0
votes
0answers
13 views

Clustering as a method to find and label classes for supervised learning

I'm working on a text classification project. We have around 300k documents (small, 1~2 phrases) and we don't know the set of labels or how many labels there are. The recommended approach to me was ...
0
votes
0answers
22 views

Can cosine similarity be used to measure similarity between words?

In text mining books, I generally see cosine similarity used as a way to assess the similarity in documents; however, by transposing a tf-idf matrix, one can also calculate cosine similarity between ...
2
votes
1answer
42 views

Why does Naive Bayes work better when the number of features >> sample size compared to more sophisticated ML algorithms?

According to this article Because of the class independence assumption, naive Bayes classifiers can quickly learn to use high dimensional features with limited training data compared to more ...
1
vote
0answers
10 views

What is an appropriate Evaluation Metric and corresponding Loss function which best optimize the metric for a classification based FAQ Chatbot?

I am developing a FAQ chatbot to display/return only one correct answer in a chat window for a given question from the user. I know MRR & MAP make sense as an evaluation metric for information ...
0
votes
1answer
53 views

Applying PCA - First two components explain low variance but have high data separation when plotting

Applying PCA on a set of documents gives strange results in terms of the variance explained by the PCs vs the data separation I'm having when plotting the first two principle components. Details: ...
0
votes
0answers
31 views

How to implement topic modelling in regression analysis

I have a dataset consisting of hotel reviews, ratings, and other features such as traveller type, and word count of the review. I want to perform topic modeling (LDA) and use the topics derived from ...
0
votes
1answer
22 views

Text Similarity - Cosine - Control. Suggestion to another / better method?

I would like to ask you, if anybody could check my code, because it was behaving weird - not working, giving me errors to suddenly working without changing anything - the code will be at the bottom. ...
0
votes
0answers
12 views

Unsupervised learning with DNN on text

I want to: Have a key:pair database with author:largetextfileofeverysentencetheauthorpublished.txt Set up a deep neural network to see without supervision patterns in choice of vocabulary. Have the ...
0
votes
1answer
31 views

Comparison of text data for same distribution

I have two datasets with different text in it. I want to check if they are from the same distribution. If they were numbers tabulated in some way, it would be easy. Since this is text, how can this be ...
0
votes
0answers
19 views

Aggregation across sentences in a document

I'm using deepMoji for a text classification problem. Deepmoji returns a vector of 64 emoji for twitter sized text. I'm running deepMoji on documents that are much longer and I'm wondering if there ...
0
votes
0answers
27 views

Current SOTA in text classification

I've been recently starting with text classification, yet the amount of work in this field is somewhat overwhelming. Could you please direct me towards some of the SOTA (deep) approaches for solving ...
0
votes
0answers
9 views

Enrich true signal from loads of data

Background For simplicity let's say we have an alphabet of ABC and we are looking at words that all have the same length (n = 10)...
1
vote
0answers
28 views

HDP: Gibbs sampler implementation

I am trying to recreate the model proposed by Gao et al. (2011), based on the Hierarchical Dirichlet Process proposed by Teh and al. (2005). To estimate the model (let's call it iHDP) I need to ...
0
votes
0answers
16 views

How do I normalize or weight document-feature-matrix by length of dictionary entries

What is the best practice way to normalize or weight document-feature-matrices by the length of dictionary entries. Here is some sample code. In reality, my example works with different issues e.g. ...
7
votes
4answers
2k views

Check if a character string is not random

Background Let's say we have an alphabet of A,B, C, D, then we look through some data and find a "word" which is DDDDDDDDCDDDDDD ...
0
votes
0answers
24 views

How do document lengths affect Gaussian Naive Bayes?

I'm trying to understand Gaussian Naive Bayes. I am training on a pre-processed subset of the 20 Newsgroup data. Each observation is around 500 attributes (words), and 1 class (of 5 possible). I ...
2
votes
1answer
57 views

Word2Vec and PyTorch - am I approaching this correctly?

My understanding of Word2Vec is that the library allows for generation of an array of numbers that approximates the meaning of a word relative to others in a sentence. My use of Word2Vec e.g. ...
0
votes
1answer
30 views

Clustering short messages

I have a dataset of short message conversations (from 1 to 20 words). I would like to cluster the messages that were sent to me to extract the different topics that were discussed by my interlocutors. ...
0
votes
1answer
8 views

text mining - vocabulary size very large

Question: when you have create a corpus of let’s say, 10,000 documents, and the vocabulary size made for these is let’s say, 1 million, what best practices exist to either work with this type of ...
0
votes
0answers
26 views

Word embedding as input or raw text?

I'm trying to implement a neural network for text recognition and I'm a little bit confused about text inputs. The goal of the network is to classify a comment, toy example: ...
0
votes
0answers
18 views

vectorization based on sentence meaning

Does anyone know a method to vectorize a sentence such that it's vector represents something related to meaning of the sentence. For example, consider the following instance.. Suppose we have two ...
1
vote
0answers
16 views

Associate keyword from textrank algorithm in R to verbatim [closed]

I have created a frequency table using TextRank algorithm in R. While, I get the frequencies, I want to associate back the text to the verbatim in the form of a matrix so that I know from where each ...
1
vote
1answer
19 views

Group of word representations

For word representation baseline people use bag-of-words or word embedding. Here, I want to understand all approaches that can be used for word representations. For example: -Bag-of-words (tfidf, n-...
0
votes
1answer
30 views

Representation for meaning of a text document

Does anyone know a technique to represent meaning of a given text document, so that two documents having same meaning will have same representations? Note: Typical systems like vectorization methods ...
3
votes
2answers
53 views

Intuition behind word vector representations

How is it possible for a vector space to represent words so that it is coincident to our intuition of words? What does the orthogonality concept in such a space mean precisely? I think we can present ...
0
votes
0answers
28 views

Sequential Long-Text Classification with Recurrent and Convolutional Neural Networks

I am thinking to build a model for predicting events from news. Before I start this task I wanted to ask if someone have tried to build something like in the link(https://arxiv.org/pdf/1603.03827.pdf) ...
0
votes
1answer
27 views

Content-based document Vectorization technique

Does anyone know a content-based vectorization scheme can be used in Natural Language Processing to represent documents in vector space??
2
votes
1answer
256 views

Tf-idf for text classification: On what should IDF be calculated?

The TF-IDF value of a word specifies how important a word for each document is. My setting is any text classification where one has multiple documents of with different classes: Let's take a lot of ...
0
votes
0answers
22 views

ANOVA on TF-IDF scores?

I'm analyzing three categories of texts, and looking to use TF-IDF scores to highlight differences in the importance of different words between the three categories. Would it be acceptable to conduct ...
0
votes
0answers
56 views

TFIDF Weighting With Multiple Categorized Documents

I'm doing keyword extraction on about 300.000 documents. The documents are job advertisement. Im doing; 1- Preprosess the job description 2- Use sklearn tfidfVectorizer with min 1 max 3 ngrams ...
0
votes
0answers
3 views

Understanding formulas in Guo et al. paper regarding end-to-end text classification

I finished reading End-to-End Multi-View Networks for Text Classification by Guo et al. (2017). The paper is only a few pages long and it's fairly intuitive. However, the equations are not described ...
1
vote
0answers
23 views

Unsupervised Text Clustering Project: How to get started? [closed]

I work for a manufacturing company where robust databases and data integrity have not always been a priority. I have a very messy and finite list of 13,102 tool descriptions. I need to find out how ...
-1
votes
1answer
27 views

How to weight features when doing text mining?

I have a case where I'm doing text mining over a list of product titles. In particular I want to run a clustering algorithm. But I also have some information about those products that I think can add ...
1
vote
0answers
12 views

Matrix of probabilities of unsupervised labeling vs annotations [closed]

Given a list of texts, annotated by topic (each text can have multiple topics), and the document-topic matrix output of an LDA (Latent Dirichlet Allocation), unsupervised model. For example: ...
0
votes
0answers
8 views

How do you find most important ngrams when the frequencies are not that diverse and reflective of information/importance?

When most "frequent" is not necessarily most "important". And when frequencies are not reflective of importance (most ngrams appear just once, twice or at best thrice) - how do you find the most "...
2
votes
0answers
69 views

R or Python Equivalent to SAS Text Miner [closed]

I am working on a project where a former member of the team used SAS Text Miner (https://support.sas.com/documentation/onlinedoc/txtminer/14.1/tmref.pdf) to complete some text mining. Unfortunately, ...
1
vote
0answers
100 views

Practically speaking, is the TF-IDF threshold universal across different corpus?

I would like to know the practical threshold of the TF-IDF (just like the practical p-value cutoff of 0.1 or 0.05 in hypothesis tests). I tried to look at it in some previous post, and some people ...
3
votes
0answers
155 views

Delta TF-IDF right choice for multi classification problem

In the paper of Martineau & Finin they describe their new approach with Delta TF-IDF . Instead of measuring how rare features are in the document, they weight these values by how biased they are ...
0
votes
0answers
22 views

can focal loss function works for text classification problem?

I am working on a relation extraction and classification problem. The data is in the form of text files. The data is imbalanced. I want to use focal loss function to address class imbalance problem in ...
0
votes
0answers
14 views

Use topic modeling to determine the topic popularity

I have a collection of documents. My end goal is to determine the popularity of the topic discussed by each document. In other words, whether (or to what extent) one document is discussing a popular ...
1
vote
1answer
28 views

How does clustering improve a language model?

This article describes a hierarchical clustering algorithm which clusters the words within a vocabulary based on their similarity, in order to improve a language model (in the article, n-grams). How ...
0
votes
1answer
20 views

natural language processing analysis

I have selected SMS Spam Collection as my dataset for natural language processing task. I have done many pre-processing tasks on dataset such as removing punctuations, spell correction, stemming, and ...
0
votes
0answers
14 views

Can we use precision and recall to evaluate text prediction model results?

I am using an LSTM model to predict/complete words from a seed input. These words or tokens represent xml markups or fields. So, as an example, I give my model a seed input : ...
1
vote
0answers
31 views

why LSTM word based model works better with random input seed than fixed input seed?

I have implemented an LSTM model that have 2 LSTM layers, a dropout layer and a dense layer for predictions. I trained my LSTM model on 1000 XML files. Each file has 4 main markups with very simple ...