Stack Exchange Network

Stack Exchange network consists of 175 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [tf-idf]

In information retrieval, tf–idf, short for term frequency–inverse document frequency, is a numerical statistic that is intended to reflect how important a word is to a document in a collection or corpus.

0
votes
0answers
7 views

How to validate results for TF-IDF

I am currently using TF-IDF and cosine similarity for document comparison. I am getting some results that, on face value don't make sense, more specifically when looking through the document I don't ...
0
votes
0answers
13 views

What's the effect of using TF-IDF encoded instead of one-hot encoded categorical data as input to a neural network?

As input into a simple neural network multi-class classifier, I am considering using a variation of the standard one-hot sparse matrix to represent categorical variables. Instead of each element ...
0
votes
0answers
7 views

Use raw count, Term Frequency or Log Normalized with Cosine Similarity?

For the task of finding duplicate documents within a corpus of documents I have been using a TF-IDF vectorizer (from sklearn in Python) combined with the Cosine Similarity between the document vectors....
0
votes
0answers
13 views

Applying TF/IDF to non-text data?

I have a classification problem in which I am supposed to predict the end state of an object based on a set of events it experiences. There are about one thousand possible events and each object is ...
0
votes
0answers
22 views

Improve the accuracy of semantic text matching

I have a corpus of ~200K sentences of variable length, the median length is 16 words. My goal is for a given sentence to find other sentences with a similar meaning. I tried several approaches: using ...
0
votes
0answers
31 views

Why does 4-gram work better than trigram or bigram or unigram in my experiments?

In a binary classification task, I used Logistic regression, decision tree and Adaboost with decision tree (max_depth=1). For each of the machine learning task, I used GridSearchCV to choose the ...
1
vote
2answers
77 views

Determining epsilon for DBSCAN

I'm using the method described in this paper for determining the optimal epsilon value for DBSCAN clustering in which a plot of the nearest neighbors is used: However, the plots in the paper and ...
0
votes
0answers
28 views

Kolmogorov–Smirnov test on text data

The Kolmogorov–Smirnov test a very efficient way to determine if two samples are significantly different from each other or whether the CDF between two different samples fit each other. This can be ...
0
votes
1answer
20 views

Did my text data come from two distinct distributions?

I have labeled text data from two different classes. I have calculated tfidf feature representation of all the sentences in question. I have a huge matrix where rows are sentences and columns are ...
0
votes
1answer
46 views

High Precision and low recall score for TF-IDF when using KNN algorithm

I have twitter data which is labelled with the sentiment(Postive, Negative, Neutral) and I have evaluated the performance of Tf-Idf and Doc2Vec feature extractor using the KNN algorithm and logistic ...
0
votes
1answer
23 views

Correct algorithm for string classification

I have a long list of DNA strings (of equal length) made of 4 letters (A,T,G,C). I want to do a binary classification on the strings. I have two basic quetsions: I have a lot of duplicate strings ...
1
vote
0answers
32 views

Clustering algorithms for multiple features that are arrays/lists

my issue is that I want to cluster data from a specific context. The dataset was 3 different texts that cannot be concatenated. Now I have the following "data-object" in Python: It is an array with ...
0
votes
0answers
42 views

What is the maximum number of features in Logistic Regression Problem

I was doing Text classification(binary) hosted on kaggle with approx 1.3 millions observations. My approach is to use Logistic Regression after computing the TF-IDF matrix with n-grams = 1:3. With ...
0
votes
1answer
30 views

Negative values in word vectorizations

I am currently in the middle of reading Applied Text Analysis with Python by Bengfort, Bilbro, and Ojeda, and encountered a sentence that I've struggled to wrap my head around. In the section ...
0
votes
0answers
18 views

Which is the best strategy for clustering a glossary of terms by exploiting their definitions

I have a glossary (dictionary) of terms together with short definitions (1 to 20 words). I want to cluster these terms by their application domains by using their definitions; for example, clustering ...
0
votes
0answers
58 views

Applying Label From Supervised Learning to Unlabeled Data- Text Classification

I am wondering if anyone has code to following: 1) Apply labels from a previous text classification dataset like this type of data (https://colab.research.google.com/drive/...
0
votes
0answers
46 views

Filter n-grams of different lengths coming from the same instance

I tokenize two documents into unigrams, bigrams and trigrams, then merge the resulting n-gram lists per document. Then, I assign a tf-idf weight to these n-grams and keep the top 30 n-grams in terms ...
0
votes
0answers
154 views

TF-IDF Normalisation When Comparing Corpora Similarity

I'm computing the similairity between two text corpora by summarising document-wise distance in some way, wherein each document of each corpora is represented as a TF-IDF vector. My question is the ...
0
votes
1answer
75 views

Text Similarity - Cosine - Control. Suggestion to another / better method?

I would like to ask you, if anybody could check my code, because it was behaving weird - not working, giving me errors to suddenly working without changing anything - the code will be at the bottom. ...
1
vote
1answer
109 views

TF-IDF String to Vector Weka bias

For example, let's say I have a text dataset like: "words text etc",label "words text etc",label "words text etc",label If I use Weka's String to Word ...
0
votes
1answer
435 views

How to combine two tfidf sparse vectors

Say that I have two document collections that I have created a tf-idf sparse vector for each one using TfidfVectorizer. How could I combine those two vectors into one that would resemble the tfidf of ...
2
votes
1answer
510 views

Tf-idf for text classification: On what should IDF be calculated?

The TF-IDF value of a word specifies how important a word for each document is. My setting is any text classification where one has multiple documents of with different classes: Let's take a lot of ...
0
votes
0answers
29 views

ANOVA on TF-IDF scores?

I'm analyzing three categories of texts, and looking to use TF-IDF scores to highlight differences in the importance of different words between the three categories. Would it be acceptable to conduct ...
0
votes
0answers
212 views

TFIDF Weighting With Multiple Categorized Documents

I'm doing keyword extraction on about 300.000 documents. The documents are job advertisement. Im doing; 1- Preprosess the job description 2- Use sklearn tfidfVectorizer with min 1 max 3 ngrams ...
0
votes
1answer
126 views

How to compute k-means on provided inverted index using tf-idf

I built an inverted index to represent the following sample documents: ...
1
vote
0answers
211 views

Practically speaking, is the TF-IDF threshold universal across different corpus?

I would like to know the practical threshold of the TF-IDF (just like the practical p-value cutoff of 0.1 or 0.05 in hypothesis tests). I tried to look at it in some previous post, and some people ...
3
votes
0answers
232 views

Delta TF-IDF right choice for multi classification problem

In the paper of Martineau & Finin they describe their new approach with Delta TF-IDF . Instead of measuring how rare features are in the document, they weight these values by how biased they are ...
0
votes
0answers
130 views

Why is the size of fitted truncated svd model is so big?

I have a dataset with tfidf matrix of shape (200000, 565000). I am fitting truncated svd of 500 dimensions from sklearn onto it and pickling the resulting svd object for later use. The pickle file is ...
0
votes
0answers
23 views

Positional similarity

What is a good positional similarity measure? Bottom line - words, which are next to each other should be ranked higher (no semantic or fuzzy matching needed). I want to use this in addition to TF/IDF-...
0
votes
0answers
147 views

How to build a User Profile in Recommendation Systems?

I am building a recommendation system using tf-idf technique and cosine similarity. So far I am already recommending items given an item information. My concern is more about the learning part. I know ...
0
votes
1answer
228 views

How to improve performance for LDA?

I am running LDA on health-related data. Specifically I have ~500 documents that contain interviews that last around 5-7 pages. Other than that, I cannot really go into the details of the data due to ...
2
votes
1answer
1k views

TF-IDF vs just TF in text classification [duplicate]

It's common to see people using a tf-idf representation of words for text classification, but I don't understand why not just use tf. Say $tf(t,d)$ is the $tf$ of term $t$ in a document $d$ and $idf(...
1
vote
0answers
331 views

Text classification using tf idf

I am developing a text classification model. At this moment I have to classify some documents in six different classes. I am using a simple approach as a starting point based on random forests over a ...
0
votes
1answer
56 views

Using k fold cross validation gives lower results than without using it

I have implemented text classification in the sentence level by following through this tutorial. I have used tf-idf and NB & SVM as shown in the tutorial. The code is working fine with my dataset. ...
1
vote
0answers
99 views

How to compute tf-idf for sequential k means?

I am trying to run the sequential K means algorithm as described here on a corpus using tf-idf as a vectorized representation of my documents. I do this because I don't have access to all of my ...
0
votes
2answers
151 views

Use tf-idf for QnA

I have a corpora of queries and answers where the queries are the title of different documents and the answers are a short description of them. E.g: Title: African Civilian Deaths Description:...
0
votes
0answers
389 views

Accuracy increases on decreasing the percentage of training data with stable precision, recall and F-score

I am currently working on a classification problem using tf-idf and Naive Bayes for two classes A and B. I have randomly shuffle the dataset before implementation, and I was experimenting with the ...
1
vote
0answers
64 views

TF-IDF on a sub-corpus

I'm analyzing newspaper articles, some are 2 or 3 pages long, some a few lines long. My Corpus is made of a few thousands of dated articles spanning over a few ...
2
votes
1answer
1k views

How does correlation work in text mining?

When we are done in pre-processing the text (removing stop words, stemming, lower case) then creating a term document that has TF-IDF as weight. How can I decide what correlation test to do between ...
3
votes
1answer
3k views

What does word embedding weighted by tf-idf mean?

The paper that I am reading explains about how it implemented the feature vector used for a twitter sentiment classification task. The first is a simple combination, where each tweet is represented ...
0
votes
0answers
34 views

How TF is calculated?

Im really confused about how TF is calculated. I understand that TF = count of term x/ total # of terms For example: ...
1
vote
2answers
2k views

Understanding and interpreting the output of Spark's TF-IDF implementation

I am currently trying to understand what the example code provided as part of Spark's TF-IDF implementation is doing. Given the example code block (taken from Spark's Github repository) ...
0
votes
1answer
926 views

How to deal with TFIDF for test classification?

I am working on a project where I have to classify texts. For that, I am dividing my data into training and test data. In order to train my classifier(to be determined later), I am planning to ...
0
votes
1answer
432 views

Finding the top words in a text

(I have no actual background in data science or statistics, so please easy with the math and concept names). I have a text file (lets say 40k words), and I want to find the top 10 words with the ...
1
vote
1answer
478 views

Does triplet loss help for document similarity search?

I build CNN network over documents with triplet loss. And compare documents with cosine similarity. It does really find similar docs and catches interesting dependencies. But simple tfidf model does ...
11
votes
2answers
6k views

How is the .similarity method in SpaCy computed?

Not Sure if this is the right stack site, but here goes. How does the .similiarity method work? Wow spaCy is great! Its tfidf model could be easier, but w2v with only one line of code?! In his 10 ...
2
votes
1answer
416 views

Classifier accuracy decreases as n of n-gram models increases. Is this expected?

I am trying to tackle a multi-classification problem that requires text processing. The data contains a lot of samples (approximately 100.000 samples) and one of the features I need to work with is a ...
2
votes
0answers
163 views

Representative terms per clusters based on tf-idf

I have a result of text clustering based on TF-IDF. I have $k$ clusters. How can I get the representative terms for each cluster $I=1,\dots,k$ using the TF-IDF matrix? Is there any standard way to do ...
1
vote
0answers
6 views

Ranking Text Posts based on NER Tags

I have a list of NER tags ranked by the number of times they appear in a database of articles, with the highest count listed first. I am trying to figure out how to score the articles based on: How ...
0
votes
1answer
115 views

Normalizing the output of your rf-idf matrix

I am following this excellent tutorial: http://blog.christianperone.com/2011/10/machine-learning-text-feature-extraction-tf-idf-part-ii/ However there is one part that I do not get: If the output of ...