Questions tagged [cosine-similarity]
An angular-type similarity coefficient between two vectors. It is like correlation, only without centering the vectors.
0
votes
0answers
7 views
Dimensionality reduction before clustering cosine data values causes a change of scale
In my experiment, I am doing hierarchical agglomerative clustering of texts (parameters: cosine, average). My features matrix is very sparse, so I considered PCA as dimensionality reduction technique. ...
0
votes
0answers
33 views
How to evaluate similarity metric using classifiers and clustering techniques?
I was going through this paper which proposes a new similarity metric. The evaluation is carried out using various classification and clustering techniques. I was confused about how a similarity ...
0
votes
0answers
16 views
Clustering/Similarity between drivers
I have a dataset that contains initial and ending points of car trips:
...
0
votes
1answer
19 views
Did my text data come from two distinct distributions?
I have labeled text data from two different classes. I have calculated tfidf feature representation of all the sentences in question. I have a huge matrix where rows are sentences and columns are ...
0
votes
0answers
12 views
Set similarity as weight to ratings
I have a problem deciding which similarity function to use.
I want to find the similarity between the users based on their requirements about computer performance metrics normalized to 1. Each user ...
0
votes
0answers
8 views
Finding Semantically Similar Learned Features
I have learned features from text and image and are projected in a hyperspace. Once I have the feature space, I am looking to find those features which are similar to each other. I have tried ...
0
votes
0answers
20 views
Cosine similarity matrix of linearly transformed inputs
Given a matrix $\mathbf{C}$ which contains pairwise cosine similarities between rows of a matrix $\mathbf{A}$, linearly transformed by matrix $\mathbf{U}$:
$$
\mathbf{C} = K(\mathbf{UA}, \mathbf{UA})
$...
2
votes
1answer
44 views
Distance metric with characteristics of cosine and Manhattan
I'm working on a project where I want to find similarities between groups of events. So far I have expressed groups of events as vectors of event counts and computing similarities between them. I'm ...
1
vote
1answer
38 views
Finding similar text - algorithms and evaluation
I've been asked to create a program that will rank similar texts to an input text given a collection of text.
So far I've been using a tdidf representation and cosine similarity with a lot of regex-...
0
votes
0answers
9 views
Measure the change of feature set over time
I have two matrices mat1 and mat2, the same number of columns but the different number of rows.
You can imagine that ...
0
votes
1answer
34 views
Calculating similarity between two lists: high cosine similarity, but high RMSE
I want to see how similar two datasets are, as a way to justify that they can be used in similar contexts. In practice one dataset contains manually calculated data, and the other automatically ...
0
votes
0answers
163 views
Weighted Cosine Similarity
To convert cosine similarity to weighted cosine similarity, one can use at least two approaches. But I don't know which one is better.
The first approach is to first reweight each vector and then ...
1
vote
1answer
69 views
Spectral Clustering of a skipgram model
I have a model where I'm applying Spectral Clustering to frequencies of words. My pipeline consists in TF-IDF, followed by a <...
0
votes
1answer
53 views
Text Similarity - Cosine - Control. Suggestion to another / better method?
I would like to ask you, if anybody could check my code, because it was behaving weird - not working, giving me errors to suddenly working without changing anything - the code will be at the bottom.
...
0
votes
0answers
43 views
Correlation 2D vector fields
Having multiple (hundreds) of 2D flow maps, ie vector fields how would one find statistical correlation between these?
Plotting yields, for visualization purposes only:
I am thinking about comparing ...
0
votes
1answer
29 views
Similarity index between two texts Ask Question
I'm trying to compare two vectors in a small NLP project using Python. Code doesn't make any difference since I'm using scikit-learn, but my doubts are about my calculations.
I have a query vector ...
2
votes
2answers
91 views
How to fit laplace/exponential distribution to cosine similarities?
I am a computational biologist with little experience fitting data.
I'm trying to fit a distribution of cosine similarities computed between sparse matrices. The goal is to be able use this ...
0
votes
0answers
9 views
Learning similarity metric from data
I want to measure the similarity of fastText vectors. Typically, for vector similarity, the cosine similarity is used.
I would like to learn a notion of similarity based on the labels of the tokens ...
0
votes
0answers
50 views
How to create a binary threshold from Cosine distance between 1-D arrays?
I have a graph of the Cosine distance between the question and the sentence most similar to it when there is an answer and when there is none.
I want to establish a threshold on the abscissa axis ...
2
votes
2answers
323 views
Cosine-Similarity vs non-linear measures
In NLP, people often use cosine similarity to measure how close two vector spaces are to each other. However, we know that cosine-similarity is the same thing as Pearson correlation, for centered ...
0
votes
1answer
37 views
K-means for data sets with scalar and vector objects
My question consists of two parts, both possibly closely related:
Part 1:
I have a dataset where the incoming data ($x$) will be an eigenvector ($V$) and an associated eigenvalue ($\lambda$). That ...
6
votes
3answers
2k views
Curse of dimensionality- does cosine similarity work better and if so, why? [duplicate]
When working with high dimensional data, it is almost useless to compare data points using euclidean distance - this is the curse of dimensionality.
However, I have read that using different distance ...
0
votes
1answer
58 views
Correctness of a skewed cosine similarity graph
I am currently implementing a word2vec model that uses the cosine similarity to determine the similarity between two vectors.
When plotting all the possible cosine similarities, I get the following ...
0
votes
0answers
840 views
How to find nearest neighbors using cosine similarity for all items from a large embeddings matrix?
I have an embeddings matrix of a large no:of items - of around 100k, with each embedding vector length of 100. So a matrix of size 100k x 100;
From this, I am trying to get the nearest neighbors for ...
1
vote
0answers
366 views
User-user suggestions with collaborative filtering (recommendation system)
I have a binary matrix N x N where both rows and columns represent users of a website. If matix[i,j] = 1 it means that ...
0
votes
1answer
41 views
Word vector normalisation by document size
I have a bunch of text documents of varying lengths (100k words to just thousands). I want to compare similarities of these vectors, specifically, cosine similarities.
While I understand that cosine ...
1
vote
0answers
257 views
SVD item similarity calculation
I am performing SVD on a rating matrix of Users and Items and I get 3 matrices out of which Vt provides latent feature for items. How do I compute similarities between a pair of items using these ...
0
votes
1answer
88 views
How to interpret very low similarity score of two vectors but having significant permutation test
I used cosine angle to characterize the similarity of two vectors $x_{1}$ and $x_{2}$, and then performed permutation test to evaluate the significance of the similarity. For permutation test, vector $...
1
vote
3answers
371 views
Cosine similarity indexing?
Are there any open source implementations out there that can efficiently solve the following.
I'm given a fixed set $S$ of $n$-dimensional vectors of size $N$, where $N$ is of the order of a million.
...
1
vote
0answers
43 views
In what way does the RV coefficient measure similarity?
The Pearson correlation coefficient is a cosine between two vectors. That's easy to understand but what happens when instead we look at the correlation between two matrices through the RV coefficient?
...
0
votes
0answers
178 views
Why is only Euclidean distance allowed to be used for Ward's method? [duplicate]
Using scipy, I noticed that I am allowed to use only Euclidean distance for Ward's method.
Is it because Ward's uses Error Sum of Squared?
What if I use Ward's method with cosine similarity?
Cosine ...
1
vote
1answer
33 views
Measuring simmilarity of observations (non numeric)
I have a dataset of format :
day,measurement1,measurement2
1,a,b
1,a,c
1,f,s
2,a,b
2,a,c
2,f,g
3,a,d
3,a,q
3,f,s
In this example day1 is more similar with day2 ...
1
vote
1answer
678 views
Correlation and Similarity
Suppose I have two vectors, both of which are probabilities of something (sum to 1). Under what circumstances will correlation (say Pearson corr.) and similarity (say cosine sim.) differ largely?
I ...
4
votes
1answer
1k views
Is feature normalisation needed prior to computing cosine distance?
I have a dataset of equal length feature vectors, where each vector contains around 20 features extracted from an audio file (fundamental frequency, BPM, ratios of high to low frequencies etc).
I am ...
3
votes
1answer
651 views
Cosine similarity?
Let's say I have two vectors of $1$ and $-1$, and I want to know how similar these two vectors are. Is the use of the cosine similarity coefficient, justifed in this case?
0
votes
0answers
100 views
Why low results with Content-Based recommendation?
I'm unable to get a good result with my Content-Based recommendation (very low Precision)
Items are different sort of services (Yelp DataSet).
To compute the item's profiles, I used a classical ...
0
votes
0answers
667 views
Cosine similarity and normalization
When I normalize a data set and compute the cosine similarity between the rows, the cosine similarity differs from the one without any normalization.
Say there are 4 2D vectors: (1, 1), (2, 2), (1, 2)...
0
votes
1answer
298 views
Using cosine similarity to measure similarity between uses is not correct
I have a theoretical question. I have implemented a recommender system using collaborative filtering method. There, I am using cosine similarity method to calculate similarity between two users. I ...
1
vote
0answers
180 views
Imputation: cosine similarity vs random forest
Long time lurker, first time question writer in CV, so please be kind.
Currently, I'm exploring the benefits and drawback to performing imputation with cosine similarity and with random forest.
...
0
votes
1answer
432 views
High dimensional clustering of percentage data using cosine similarity
I'm building a clustering algorithm and was trying to determine the best way to get separate and accurate clusters.
I have 300+ features to cluster on, and they are all percentages between 0 and 1. ...
1
vote
2answers
2k views
Drawbacks with Cosine Similarity
I am assessing the similarity between documents represented as vectors of tf-idf values. I know that the cosine similarity is a well-defined and commonly used measure in information retrieval.
...
1
vote
1answer
633 views
what are some clustering algorithms that work with cosine similarity as distance measure?
I am trying to find clustering algorithms which can work with cosine similarity for tweet classification.
1
vote
1answer
2k views
Why use the cosine distance for machine translation (Mikolov paper)?
I am currently reading the paper "Exploiting Similarities among Languages for Machine Translation" by Mikolov et al. (available here : https://static.googleusercontent.com/media/research.google.com/en/...
1
vote
0answers
108 views
does dimensionality reduction work in similarity measures?
I'm performing classification via cosine similarity to vector means. Normally, we reduce dimensionality of a problem in order to reduce confusion to the classifier. Mathematically, will ...
1
vote
0answers
74 views
How to teach neural network to get meaning similarity measure for 2 texts
I have an NLP task: there are database with 2 bunch of texts (50000 strings)
1) customer complaints (i.e. "i have a problem with authorization system ...")
2) answers to this complaints (i.e. "Go to ...
1
vote
1answer
485 views
Cosine Similarity Intuition
I understand what cosine similarity is and how to calculate it, specifically in the context of text mining (i.e. comparing tf-idf document vectors to find similar documents). What I'm looking for is ...
1
vote
0answers
149 views
Supervised cosine similarity
Suppose we have some samples, each sample is with two vectors and the corresponding label. That is, it looks like ($\mathbf{u}_i, \mathbf{v}_i, y_i$), where $y_i \in \{0, 1\}$
We can calculate the ...
0
votes
1answer
569 views
Item-item similarity using adjusted cosine / Pearson correlation
I'm following a lecture that explains how to calculate item-item similarities using adjusted cosine distance (or Pearson correlation). I tried implementing this and have not gotten the same results.
...
0
votes
1answer
66 views
What is the best similarity metric for bottleneck features of images extracted from a pre-trained neural network?
I have a bunch of images and a pre-trained AlexNet model. I have fetched bottleneck features for all images as their representations. I want to find top-K similar images for a given image. What ...
3
votes
1answer
583 views
Cosine distance with latitude and longitude
I have several features I'd like to use for computing cosine similarity between rows in a data set. However, two of them are latitude and longitude.
Apart from the fact that it's not the "correct" ...
