Questions tagged [cosine-similarity]
An angular-type similarity coefficient between two vectors. It is like correlation, only without centering the vectors.
89
questions
0
votes
0answers
11 views
Deep item-based recommender objective function
I'm trying to understand the following paper written by researchers at eBay that uses deep learning to overcome the problem of making recommendations when you mostly have one-of-a-kind items. A ...
0
votes
0answers
23 views
Extension of the K-Nearest neighbor algorithm to get results in different neighborhoods
I would like to use the kNN algorithm to find the closest neighbor to a vector. But I would like it to limit to a point per neighborhood (radius)
In this image, given the point in red, I would like ...
0
votes
0answers
32 views
Should I normalize the vectors by row or by column before performing cosine similarity?
I have a dataset which contains vectors of different features that generated from subtitles in movies, something like:
...
1
vote
0answers
20 views
How can I define a accuracy measure for word2vec predictions
I have a data set consisting of tags and some classes.I'm suppose to find the nearest class to each set of tags with Word2vec embeddings and cosine similarity.Each set of tags have multiple classes ...
1
vote
0answers
25 views
the accuracy of covariance between two high-dimensional vectors
Question
Is the covariance between high-dimensional vectors less accruate than covariance between two vectors in low-dimensional vecotrs?
I am asking this questio to check if there is a need for '...
0
votes
0answers
18 views
analyze time-series consine similarity matrices (in R)
I have calculated the cosine similiarity between multiple users for 10 years separately.
I can visualize and cluster the matrix of each year but I was wondering if there is a smart approach, vignette ...
0
votes
0answers
5 views
Dispersion index and Cosine distance
I have different sets of vectors of dimension n and I want to measure and compare the dispersion index of each set.
For my specific problem, each set contains vector representations of words (...
0
votes
0answers
15 views
Time complexity for locality sensitive hashing similar image search
I am trying to find most visually similar images for large image dataset. (N=1 million), using LSH (Locality Sensitive Hashing).
Image feature vectors are 4096 dimensional VGG-16 features.
Now, my ...
0
votes
0answers
15 views
How to validate results for TF-IDF
I am currently using TF-IDF and cosine similarity for document comparison. I am getting some results that, on face value don't make sense, more specifically when looking through the document I don't ...
2
votes
0answers
43 views
Are there advanced “cos similarity” that influence of dim size is less?
I found that the cosine similarity is
affected to the effect of "Curse of dimension" by trying the following simulation.
create(select) two vectors form uniform random numbers U[-1, 1], each dim = 2,...
0
votes
0answers
13 views
Dimensionality reduction before clustering cosine data values causes a change of scale
In my experiment, I am doing hierarchical agglomerative clustering of texts (parameters: cosine, average). My features matrix is very sparse, so I considered PCA as dimensionality reduction technique. ...
0
votes
0answers
36 views
How to evaluate similarity metric using classifiers and clustering techniques?
I was going through this paper which proposes a new similarity metric. The evaluation is carried out using various classification and clustering techniques. I was confused about how a similarity ...
0
votes
0answers
26 views
Clustering/Similarity between drivers
I have a dataset that contains initial and ending points of car trips:
...
0
votes
1answer
21 views
Did my text data come from two distinct distributions?
I have labeled text data from two different classes. I have calculated tfidf feature representation of all the sentences in question. I have a huge matrix where rows are sentences and columns are ...
0
votes
0answers
33 views
Set similarity as weight to ratings
I have a problem deciding which similarity function to use.
I want to find the similarity between the users based on their requirements about computer performance metrics normalized to 1. Each user ...
0
votes
0answers
11 views
Finding Semantically Similar Learned Features
I have learned features from text and image and are projected in a hyperspace. Once I have the feature space, I am looking to find those features which are similar to each other. I have tried ...
0
votes
0answers
22 views
Cosine similarity matrix of linearly transformed inputs
Given a matrix $\mathbf{C}$ which contains pairwise cosine similarities between rows of a matrix $\mathbf{A}$, linearly transformed by matrix $\mathbf{U}$:
$$
\mathbf{C} = K(\mathbf{UA}, \mathbf{UA})
$...
2
votes
1answer
56 views
Distance metric with characteristics of cosine and Manhattan
I'm working on a project where I want to find similarities between groups of events. So far I have expressed groups of events as vectors of event counts and computing similarities between them. I'm ...
1
vote
1answer
132 views
Finding similar text - algorithms and evaluation
I've been asked to create a program that will rank similar texts to an input text given a collection of text.
So far I've been using a tdidf representation and cosine similarity with a lot of regex-...
0
votes
0answers
10 views
Measure the change of feature set over time
I have two matrices mat1 and mat2, the same number of columns but the different number of rows.
You can imagine that ...
0
votes
1answer
62 views
Calculating similarity between two lists: high cosine similarity, but high RMSE
I want to see how similar two datasets are, as a way to justify that they can be used in similar contexts. In practice one dataset contains manually calculated data, and the other automatically ...
1
vote
0answers
548 views
Weighted Cosine Similarity
To convert cosine similarity to weighted cosine similarity, one can use at least two approaches. But I don't know which one is better.
The first approach is to first reweight each vector and then ...
1
vote
1answer
96 views
Spectral Clustering of a skipgram model
I have a model where I'm applying Spectral Clustering to frequencies of words. My pipeline consists in TF-IDF, followed by a <...
0
votes
1answer
89 views
Text Similarity - Cosine - Control. Suggestion to another / better method?
I would like to ask you, if anybody could check my code, because it was behaving weird - not working, giving me errors to suddenly working without changing anything - the code will be at the bottom.
...
0
votes
0answers
72 views
Correlation 2D vector fields
Having multiple (hundreds) of 2D flow maps, ie vector fields how would one find statistical correlation between these?
Plotting yields, for visualization purposes only:
I am thinking about comparing ...
0
votes
1answer
33 views
Similarity index between two texts Ask Question
I'm trying to compare two vectors in a small NLP project using Python. Code doesn't make any difference since I'm using scikit-learn, but my doubts are about my calculations.
I have a query vector ...
2
votes
2answers
171 views
How to fit laplace/exponential distribution to cosine similarities?
I am a computational biologist with little experience fitting data.
I'm trying to fit a distribution of cosine similarities computed between sparse matrices. The goal is to be able use this ...
0
votes
0answers
14 views
Learning similarity metric from data
I want to measure the similarity of fastText vectors. Typically, for vector similarity, the cosine similarity is used.
I would like to learn a notion of similarity based on the labels of the tokens ...
0
votes
0answers
54 views
How to create a binary threshold from Cosine distance between 1-D arrays?
I have a graph of the Cosine distance between the question and the sentence most similar to it when there is an answer and when there is none.
I want to establish a threshold on the abscissa axis ...
2
votes
2answers
480 views
Cosine-Similarity vs non-linear measures
In NLP, people often use cosine similarity to measure how close two vector spaces are to each other. However, we know that cosine-similarity is the same thing as Pearson correlation, for centered ...
0
votes
1answer
61 views
K-means for data sets with scalar and vector objects
My question consists of two parts, both possibly closely related:
Part 1:
I have a dataset where the incoming data ($x$) will be an eigenvector ($V$) and an associated eigenvalue ($\lambda$). That ...
7
votes
3answers
3k views
Curse of dimensionality- does cosine similarity work better and if so, why? [duplicate]
When working with high dimensional data, it is almost useless to compare data points using euclidean distance - this is the curse of dimensionality.
However, I have read that using different distance ...
0
votes
1answer
80 views
Correctness of a skewed cosine similarity graph
I am currently implementing a word2vec model that uses the cosine similarity to determine the similarity between two vectors.
When plotting all the possible cosine similarities, I get the following ...
0
votes
0answers
944 views
How to find nearest neighbors using cosine similarity for all items from a large embeddings matrix?
I have an embeddings matrix of a large no:of items - of around 100k, with each embedding vector length of 100. So a matrix of size 100k x 100;
From this, I am trying to get the nearest neighbors for ...
1
vote
0answers
443 views
User-user suggestions with collaborative filtering (recommendation system)
I have a binary matrix N x N where both rows and columns represent users of a website. If matix[i,j] = 1 it means that ...
0
votes
1answer
74 views
Word vector normalisation by document size
I have a bunch of text documents of varying lengths (100k words to just thousands). I want to compare similarities of these vectors, specifically, cosine similarities.
While I understand that cosine ...
1
vote
0answers
321 views
SVD item similarity calculation
I am performing SVD on a rating matrix of Users and Items and I get 3 matrices out of which Vt provides latent feature for items. How do I compute similarities between a pair of items using these ...
0
votes
1answer
137 views
How to interpret very low similarity score of two vectors but having significant permutation test
I used cosine angle to characterize the similarity of two vectors $x_{1}$ and $x_{2}$, and then performed permutation test to evaluate the significance of the similarity. For permutation test, vector $...
2
votes
3answers
528 views
Cosine similarity indexing?
Are there any open source implementations out there that can efficiently solve the following.
I'm given a fixed set $S$ of $n$-dimensional vectors of size $N$, where $N$ is of the order of a million.
...
1
vote
0answers
55 views
In what way does the RV coefficient measure similarity?
The Pearson correlation coefficient is a cosine between two vectors. That's easy to understand but what happens when instead we look at the correlation between two matrices through the RV coefficient?
...
0
votes
0answers
280 views
Why is only Euclidean distance allowed to be used for Ward's method? [duplicate]
Using scipy, I noticed that I am allowed to use only Euclidean distance for Ward's method.
Is it because Ward's uses Error Sum of Squared?
What if I use Ward's method with cosine similarity?
Cosine ...
1
vote
1answer
35 views
Measuring simmilarity of observations (non numeric)
I have a dataset of format :
day,measurement1,measurement2
1,a,b
1,a,c
1,f,s
2,a,b
2,a,c
2,f,g
3,a,d
3,a,q
3,f,s
In this example day1 is more similar with day2 ...
1
vote
1answer
862 views
Correlation and Similarity
Suppose I have two vectors, both of which are probabilities of something (sum to 1). Under what circumstances will correlation (say Pearson corr.) and similarity (say cosine sim.) differ largely?
I ...
4
votes
1answer
2k views
Is feature normalisation needed prior to computing cosine distance?
I have a dataset of equal length feature vectors, where each vector contains around 20 features extracted from an audio file (fundamental frequency, BPM, ratios of high to low frequencies etc).
I am ...
2
votes
1answer
745 views
Cosine similarity?
Let's say I have two vectors of $1$ and $-1$, and I want to know how similar these two vectors are. Is the use of the cosine similarity coefficient, justifed in this case?
0
votes
0answers
107 views
Why low results with Content-Based recommendation?
I'm unable to get a good result with my Content-Based recommendation (very low Precision)
Items are different sort of services (Yelp DataSet).
To compute the item's profiles, I used a classical ...
0
votes
0answers
771 views
Cosine similarity and normalization
When I normalize a data set and compute the cosine similarity between the rows, the cosine similarity differs from the one without any normalization.
Say there are 4 2D vectors: (1, 1), (2, 2), (1, 2)...
0
votes
1answer
309 views
Using cosine similarity to measure similarity between uses is not correct
I have a theoretical question. I have implemented a recommender system using collaborative filtering method. There, I am using cosine similarity method to calculate similarity between two users. I ...
1
vote
0answers
219 views
Imputation: cosine similarity vs random forest
Long time lurker, first time question writer in CV, so please be kind.
Currently, I'm exploring the benefits and drawback to performing imputation with cosine similarity and with random forest.
...
0
votes
1answer
470 views
High dimensional clustering of percentage data using cosine similarity
I'm building a clustering algorithm and was trying to determine the best way to get separate and accurate clusters.
I have 300+ features to cluster on, and they are all percentages between 0 and 1. ...
