Questions tagged [cosine-similarity]
An angular-type similarity coefficient between two vectors. It is like correlation, only without centering the vectors.
0
votes
0answers
7 views
How to validate results for TF-IDF
I am currently using TF-IDF and cosine similarity for document comparison. I am getting some results that, on face value don't make sense, more specifically when looking through the document I don't ...
2
votes
0answers
37 views
Are there advanced “cos similarity” that influence of dim size is less?
I found that the cosine similarity is
affected to the effect of "Curse of dimension" by trying the following simulation.
create(select) two vectors form uniform random numbers U[-1, 1], each dim = 2,...
0
votes
0answers
10 views
Dimensionality reduction before clustering cosine data values causes a change of scale
In my experiment, I am doing hierarchical agglomerative clustering of texts (parameters: cosine, average). My features matrix is very sparse, so I considered PCA as dimensionality reduction technique. ...
0
votes
0answers
34 views
How to evaluate similarity metric using classifiers and clustering techniques?
I was going through this paper which proposes a new similarity metric. The evaluation is carried out using various classification and clustering techniques. I was confused about how a similarity ...
0
votes
0answers
21 views
Clustering/Similarity between drivers
I have a dataset that contains initial and ending points of car trips:
...
0
votes
1answer
20 views
Did my text data come from two distinct distributions?
I have labeled text data from two different classes. I have calculated tfidf feature representation of all the sentences in question. I have a huge matrix where rows are sentences and columns are ...
0
votes
0answers
25 views
Set similarity as weight to ratings
I have a problem deciding which similarity function to use.
I want to find the similarity between the users based on their requirements about computer performance metrics normalized to 1. Each user ...
0
votes
0answers
8 views
Finding Semantically Similar Learned Features
I have learned features from text and image and are projected in a hyperspace. Once I have the feature space, I am looking to find those features which are similar to each other. I have tried ...
0
votes
0answers
20 views
Cosine similarity matrix of linearly transformed inputs
Given a matrix $\mathbf{C}$ which contains pairwise cosine similarities between rows of a matrix $\mathbf{A}$, linearly transformed by matrix $\mathbf{U}$:
$$
\mathbf{C} = K(\mathbf{UA}, \mathbf{UA})
$...
2
votes
1answer
50 views
Distance metric with characteristics of cosine and Manhattan
I'm working on a project where I want to find similarities between groups of events. So far I have expressed groups of events as vectors of event counts and computing similarities between them. I'm ...
1
vote
1answer
58 views
Finding similar text - algorithms and evaluation
I've been asked to create a program that will rank similar texts to an input text given a collection of text.
So far I've been using a tdidf representation and cosine similarity with a lot of regex-...
0
votes
0answers
9 views
Measure the change of feature set over time
I have two matrices mat1 and mat2, the same number of columns but the different number of rows.
You can imagine that ...
0
votes
1answer
46 views
Calculating similarity between two lists: high cosine similarity, but high RMSE
I want to see how similar two datasets are, as a way to justify that they can be used in similar contexts. In practice one dataset contains manually calculated data, and the other automatically ...
0
votes
0answers
377 views
Weighted Cosine Similarity
To convert cosine similarity to weighted cosine similarity, one can use at least two approaches. But I don't know which one is better.
The first approach is to first reweight each vector and then ...
1
vote
1answer
78 views
Spectral Clustering of a skipgram model
I have a model where I'm applying Spectral Clustering to frequencies of words. My pipeline consists in TF-IDF, followed by a <...
0
votes
1answer
75 views
Text Similarity - Cosine - Control. Suggestion to another / better method?
I would like to ask you, if anybody could check my code, because it was behaving weird - not working, giving me errors to suddenly working without changing anything - the code will be at the bottom.
...
0
votes
0answers
55 views
Correlation 2D vector fields
Having multiple (hundreds) of 2D flow maps, ie vector fields how would one find statistical correlation between these?
Plotting yields, for visualization purposes only:
I am thinking about comparing ...
0
votes
1answer
31 views
Similarity index between two texts Ask Question
I'm trying to compare two vectors in a small NLP project using Python. Code doesn't make any difference since I'm using scikit-learn, but my doubts are about my calculations.
I have a query vector ...
2
votes
2answers
134 views
How to fit laplace/exponential distribution to cosine similarities?
I am a computational biologist with little experience fitting data.
I'm trying to fit a distribution of cosine similarities computed between sparse matrices. The goal is to be able use this ...
0
votes
0answers
10 views
Learning similarity metric from data
I want to measure the similarity of fastText vectors. Typically, for vector similarity, the cosine similarity is used.
I would like to learn a notion of similarity based on the labels of the tokens ...
0
votes
0answers
52 views
How to create a binary threshold from Cosine distance between 1-D arrays?
I have a graph of the Cosine distance between the question and the sentence most similar to it when there is an answer and when there is none.
I want to establish a threshold on the abscissa axis ...
2
votes
2answers
413 views
Cosine-Similarity vs non-linear measures
In NLP, people often use cosine similarity to measure how close two vector spaces are to each other. However, we know that cosine-similarity is the same thing as Pearson correlation, for centered ...
0
votes
1answer
51 views
K-means for data sets with scalar and vector objects
My question consists of two parts, both possibly closely related:
Part 1:
I have a dataset where the incoming data ($x$) will be an eigenvector ($V$) and an associated eigenvalue ($\lambda$). That ...
6
votes
3answers
3k views
Curse of dimensionality- does cosine similarity work better and if so, why? [duplicate]
When working with high dimensional data, it is almost useless to compare data points using euclidean distance - this is the curse of dimensionality.
However, I have read that using different distance ...
0
votes
1answer
66 views
Correctness of a skewed cosine similarity graph
I am currently implementing a word2vec model that uses the cosine similarity to determine the similarity between two vectors.
When plotting all the possible cosine similarities, I get the following ...
0
votes
0answers
897 views
How to find nearest neighbors using cosine similarity for all items from a large embeddings matrix?
I have an embeddings matrix of a large no:of items - of around 100k, with each embedding vector length of 100. So a matrix of size 100k x 100;
From this, I am trying to get the nearest neighbors for ...
1
vote
0answers
391 views
User-user suggestions with collaborative filtering (recommendation system)
I have a binary matrix N x N where both rows and columns represent users of a website. If matix[i,j] = 1 it means that ...
0
votes
1answer
51 views
Word vector normalisation by document size
I have a bunch of text documents of varying lengths (100k words to just thousands). I want to compare similarities of these vectors, specifically, cosine similarities.
While I understand that cosine ...
1
vote
0answers
309 views
SVD item similarity calculation
I am performing SVD on a rating matrix of Users and Items and I get 3 matrices out of which Vt provides latent feature for items. How do I compute similarities between a pair of items using these ...
0
votes
1answer
107 views
How to interpret very low similarity score of two vectors but having significant permutation test
I used cosine angle to characterize the similarity of two vectors $x_{1}$ and $x_{2}$, and then performed permutation test to evaluate the significance of the similarity. For permutation test, vector $...
1
vote
3answers
433 views
Cosine similarity indexing?
Are there any open source implementations out there that can efficiently solve the following.
I'm given a fixed set $S$ of $n$-dimensional vectors of size $N$, where $N$ is of the order of a million.
...
1
vote
0answers
51 views
In what way does the RV coefficient measure similarity?
The Pearson correlation coefficient is a cosine between two vectors. That's easy to understand but what happens when instead we look at the correlation between two matrices through the RV coefficient?
...
0
votes
0answers
223 views
Why is only Euclidean distance allowed to be used for Ward's method? [duplicate]
Using scipy, I noticed that I am allowed to use only Euclidean distance for Ward's method.
Is it because Ward's uses Error Sum of Squared?
What if I use Ward's method with cosine similarity?
Cosine ...
1
vote
1answer
35 views
Measuring simmilarity of observations (non numeric)
I have a dataset of format :
day,measurement1,measurement2
1,a,b
1,a,c
1,f,s
2,a,b
2,a,c
2,f,g
3,a,d
3,a,q
3,f,s
In this example day1 is more similar with day2 ...
1
vote
1answer
770 views
Correlation and Similarity
Suppose I have two vectors, both of which are probabilities of something (sum to 1). Under what circumstances will correlation (say Pearson corr.) and similarity (say cosine sim.) differ largely?
I ...
4
votes
1answer
2k views
Is feature normalisation needed prior to computing cosine distance?
I have a dataset of equal length feature vectors, where each vector contains around 20 features extracted from an audio file (fundamental frequency, BPM, ratios of high to low frequencies etc).
I am ...
2
votes
1answer
704 views
Cosine similarity?
Let's say I have two vectors of $1$ and $-1$, and I want to know how similar these two vectors are. Is the use of the cosine similarity coefficient, justifed in this case?
0
votes
0answers
104 views
Why low results with Content-Based recommendation?
I'm unable to get a good result with my Content-Based recommendation (very low Precision)
Items are different sort of services (Yelp DataSet).
To compute the item's profiles, I used a classical ...
0
votes
0answers
722 views
Cosine similarity and normalization
When I normalize a data set and compute the cosine similarity between the rows, the cosine similarity differs from the one without any normalization.
Say there are 4 2D vectors: (1, 1), (2, 2), (1, 2)...
0
votes
1answer
306 views
Using cosine similarity to measure similarity between uses is not correct
I have a theoretical question. I have implemented a recommender system using collaborative filtering method. There, I am using cosine similarity method to calculate similarity between two users. I ...
1
vote
0answers
199 views
Imputation: cosine similarity vs random forest
Long time lurker, first time question writer in CV, so please be kind.
Currently, I'm exploring the benefits and drawback to performing imputation with cosine similarity and with random forest.
...
0
votes
1answer
452 views
High dimensional clustering of percentage data using cosine similarity
I'm building a clustering algorithm and was trying to determine the best way to get separate and accurate clusters.
I have 300+ features to cluster on, and they are all percentages between 0 and 1. ...
1
vote
2answers
3k views
Drawbacks with Cosine Similarity
I am assessing the similarity between documents represented as vectors of tf-idf values. I know that the cosine similarity is a well-defined and commonly used measure in information retrieval.
...
1
vote
1answer
779 views
what are some clustering algorithms that work with cosine similarity as distance measure?
I am trying to find clustering algorithms which can work with cosine similarity for tweet classification.
1
vote
1answer
2k views
Why use the cosine distance for machine translation (Mikolov paper)?
I am currently reading the paper "Exploiting Similarities among Languages for Machine Translation" by Mikolov et al. (available here : https://static.googleusercontent.com/media/research.google.com/en/...
1
vote
0answers
115 views
does dimensionality reduction work in similarity measures?
I'm performing classification via cosine similarity to vector means. Normally, we reduce dimensionality of a problem in order to reduce confusion to the classifier. Mathematically, will ...
1
vote
0answers
74 views
How to teach neural network to get meaning similarity measure for 2 texts
I have an NLP task: there are database with 2 bunch of texts (50000 strings)
1) customer complaints (i.e. "i have a problem with authorization system ...")
2) answers to this complaints (i.e. "Go to ...
1
vote
1answer
520 views
Cosine Similarity Intuition
I understand what cosine similarity is and how to calculate it, specifically in the context of text mining (i.e. comparing tf-idf document vectors to find similar documents). What I'm looking for is ...
1
vote
0answers
167 views
Supervised cosine similarity
Suppose we have some samples, each sample is with two vectors and the corresponding label. That is, it looks like ($\mathbf{u}_i, \mathbf{v}_i, y_i$), where $y_i \in \{0, 1\}$
We can calculate the ...
0
votes
1answer
599 views
Item-item similarity using adjusted cosine / Pearson correlation
I'm following a lecture that explains how to calculate item-item similarities using adjusted cosine distance (or Pearson correlation). I tried implementing this and have not gotten the same results.
...
