Questions tagged [cosine-similarity]
An angular-type similarity coefficient between two vectors. It is like correlation, only without centering the vectors.
97
questions
1
vote
0answers
5 views
Normalized Levenshtein distance and triangle inequality
One question regarding to the triangle inequality of normalized Levenshtein Distance. I use the well-known form D(X,Y) = 1 - d(X,Y) / MAX(|X|,|Y|) where ...
0
votes
0answers
20 views
Right way to use Cosine Similarity in R using TermDocumentMatrix
I have a TermDocumentMatrix of ~500 documents, with ~90 terms as some kind of train set. I want to implement a Cosine Similarity function, to classify a new document. It is short documents(messages), ...
1
vote
1answer
46 views
Rao-Stirling diversity index
I need help to calculate the Rao-Stirling diversity index. I tried it several times but cannot achieve equal results to R packages yet (e.g. diverse).
I used the Rao-Stirling diversity index as ...
0
votes
0answers
18 views
Cosine Similarity for Classification to EM Cluster?
Perhaps my question sounds naive, uncovering the very little knowledge that I have in the field of Statistics, but is very urgent to get a solid answer or trigger for further insights for my concerns.
...
0
votes
1answer
25 views
How to find the list of nearest vectors if ony a vector is given?
I know there are many ways to compute similarity of two different non-zero vectors but is it possible to get a list of nearest vectors whose values are continous given a single continous vector.
Lets ...
0
votes
1answer
25 views
Cosine similarity for recommendation systems
Recently picked up recommendation systems and was going through User Based Collaborative Filtering(UB-CF).
Somewhere in the text, it specified that cosine similarity is one of the measures to find ...
0
votes
1answer
41 views
Why is the cosine similarity between these (seemingly uncorrelated) vectors so high?
I am calculating the correlations between vectors of experimental data by a variety of methods (Pearson's, cosine similarity, Euclidian distance, etc.). Most the results look fine, but occasionally ...
0
votes
1answer
41 views
Cosine similarity vs quadratic objective function
The aim is to calculate the similarity between two foods given the nutritional content of each. After some reading, it seems the most popular measure for this sort of problem is the cosine similarity ...
0
votes
1answer
28 views
Does it make sense to normalize vectors after PCA for cosine distance?
I start off with word2vec embeddings and process them in the following way:
Standardize dimensions to mean 0 and standard deviation of 1
PCA to keep the top k-dimensional eigenvector, whereby ...
0
votes
0answers
32 views
What does a high cosine similarity score mean here?
I have a set of 50,000 documents divided into two classes.
Class 1 has 5000 documents and,
Class 2 has 45,000 documents
Using word2vec embeddings, I extract 300 dimensional dense, real-valued ...
1
vote
0answers
14 views
Deep item-based recommender objective function
I'm trying to understand the following paper written by researchers at eBay that uses deep learning to overcome the problem of making recommendations when you mostly have one-of-a-kind items. A ...
0
votes
0answers
38 views
Extension of the K-Nearest neighbor algorithm to get results in different neighborhoods
I would like to use the kNN algorithm to find the closest neighbor to a vector. But I would like it to limit to a point per neighborhood (radius)
In this image, given the point in red, I would like ...
0
votes
0answers
42 views
Should I normalize the vectors by row or by column before performing cosine similarity?
I have a dataset which contains vectors of different features that generated from subtitles in movies, something like:
...
1
vote
0answers
65 views
How can I define a accuracy measure for word2vec predictions
I have a data set consisting of tags and some classes.I'm suppose to find the nearest class to each set of tags with Word2vec embeddings and cosine similarity.Each set of tags have multiple classes ...
1
vote
0answers
33 views
the accuracy of covariance between two high-dimensional vectors
Question
Is the covariance between high-dimensional vectors less accruate than covariance between two vectors in low-dimensional vecotrs?
I am asking this questio to check if there is a need for '...
0
votes
0answers
22 views
analyze time-series consine similarity matrices (in R)
I have calculated the cosine similiarity between multiple users for 10 years separately.
I can visualize and cluster the matrix of each year but I was wondering if there is a smart approach, vignette ...
0
votes
0answers
6 views
Dispersion index and Cosine distance
I have different sets of vectors of dimension n and I want to measure and compare the dispersion index of each set.
For my specific problem, each set contains vector representations of words (...
0
votes
0answers
23 views
Time complexity for locality sensitive hashing similar image search
I am trying to find most visually similar images for large image dataset. (N=1 million), using LSH (Locality Sensitive Hashing).
Image feature vectors are 4096 dimensional VGG-16 features.
Now, my ...
0
votes
0answers
26 views
How to validate results for TF-IDF
I am currently using TF-IDF and cosine similarity for document comparison. I am getting some results that, on face value don't make sense, more specifically when looking through the document I don't ...
2
votes
0answers
51 views
Are there advanced “cos similarity” that influence of dim size is less?
I found that the cosine similarity is
affected to the effect of "Curse of dimension" by trying the following simulation.
create(select) two vectors form uniform random numbers U[-1, 1], each dim = 2,...
0
votes
0answers
36 views
Dimensionality reduction before clustering cosine data values causes a change of scale
In my experiment, I am doing hierarchical agglomerative clustering of texts (parameters: cosine, average). My features matrix is very sparse, so I considered PCA as dimensionality reduction technique. ...
0
votes
0answers
43 views
How to evaluate similarity metric using classifiers and clustering techniques?
I was going through this paper which proposes a new similarity metric. The evaluation is carried out using various classification and clustering techniques. I was confused about how a similarity ...
0
votes
0answers
38 views
Clustering/Similarity between drivers
I have a dataset that contains initial and ending points of car trips:
...
0
votes
1answer
22 views
Did my text data come from two distinct distributions?
I have labeled text data from two different classes. I have calculated tfidf feature representation of all the sentences in question. I have a huge matrix where rows are sentences and columns are ...
0
votes
0answers
58 views
Set similarity as weight to ratings
I have a problem deciding which similarity function to use.
I want to find the similarity between the users based on their requirements about computer performance metrics normalized to 1. Each user ...
0
votes
0answers
11 views
Finding Semantically Similar Learned Features
I have learned features from text and image and are projected in a hyperspace. Once I have the feature space, I am looking to find those features which are similar to each other. I have tried ...
0
votes
0answers
25 views
Cosine similarity matrix of linearly transformed inputs
Given a matrix $\mathbf{C}$ which contains pairwise cosine similarities between rows of a matrix $\mathbf{A}$, linearly transformed by matrix $\mathbf{U}$:
$$
\mathbf{C} = K(\mathbf{UA}, \mathbf{UA})
$...
2
votes
1answer
63 views
Distance metric with characteristics of cosine and Manhattan
I'm working on a project where I want to find similarities between groups of events. So far I have expressed groups of events as vectors of event counts and computing similarities between them. I'm ...
1
vote
1answer
361 views
Finding similar text - algorithms and evaluation
I've been asked to create a program that will rank similar texts to an input text given a collection of text.
So far I've been using a tdidf representation and cosine similarity with a lot of regex-...
0
votes
0answers
10 views
Measure the change of feature set over time
I have two matrices mat1 and mat2, the same number of columns but the different number of rows.
You can imagine that ...
0
votes
1answer
127 views
Calculating similarity between two lists: high cosine similarity, but high RMSE
I want to see how similar two datasets are, as a way to justify that they can be used in similar contexts. In practice one dataset contains manually calculated data, and the other automatically ...
3
votes
0answers
872 views
Weighted Cosine Similarity
To convert cosine similarity to weighted cosine similarity, one can use at least two approaches. But I don't know which one is better.
The first approach is to first reweight each vector and then ...
1
vote
1answer
150 views
Spectral Clustering of a skipgram model
I have a model where I'm applying Spectral Clustering to frequencies of words. My pipeline consists in TF-IDF, followed by a <...
0
votes
1answer
128 views
Text Similarity - Cosine - Control. Suggestion to another / better method?
I would like to ask you, if anybody could check my code, because it was behaving weird - not working, giving me errors to suddenly working without changing anything - the code will be at the bottom.
...
0
votes
1answer
40 views
Similarity index between two texts Ask Question
I'm trying to compare two vectors in a small NLP project using Python. Code doesn't make any difference since I'm using scikit-learn, but my doubts are about my calculations.
I have a query vector ...
2
votes
2answers
233 views
How to fit laplace/exponential distribution to cosine similarities?
I am a computational biologist with little experience fitting data.
I'm trying to fit a distribution of cosine similarities computed between sparse matrices. The goal is to be able use this ...
0
votes
0answers
62 views
How to create a binary threshold from Cosine distance between 1-D arrays?
I have a graph of the Cosine distance between the question and the sentence most similar to it when there is an answer and when there is none.
I want to establish a threshold on the abscissa axis ...
2
votes
2answers
613 views
Cosine-Similarity vs non-linear measures
In NLP, people often use cosine similarity to measure how close two vector spaces are to each other. However, we know that cosine-similarity is the same thing as Pearson correlation, for centered ...
0
votes
1answer
97 views
K-means for data sets with scalar and vector objects
My question consists of two parts, both possibly closely related:
Part 1:
I have a dataset where the incoming data ($x$) will be an eigenvector ($V$) and an associated eigenvalue ($\lambda$). That ...
8
votes
3answers
5k views
Curse of dimensionality- does cosine similarity work better and if so, why? [duplicate]
When working with high dimensional data, it is almost useless to compare data points using euclidean distance - this is the curse of dimensionality.
However, I have read that using different distance ...
0
votes
1answer
97 views
Correctness of a skewed cosine similarity graph
I am currently implementing a word2vec model that uses the cosine similarity to determine the similarity between two vectors.
When plotting all the possible cosine similarities, I get the following ...
0
votes
2answers
1k views
How to find nearest neighbors using cosine similarity for all items from a large embeddings matrix?
I have an embeddings matrix of a large no:of items - of around 100k, with each embedding vector length of 100. So a matrix of size 100k x 100;
From this, I am trying to get the nearest neighbors for ...
1
vote
0answers
502 views
User-user suggestions with collaborative filtering (recommendation system)
I have a binary matrix N x N where both rows and columns represent users of a website. If matix[i,j] = 1 it means that ...
0
votes
1answer
154 views
Word vector normalisation by document size
I have a bunch of text documents of varying lengths (100k words to just thousands). I want to compare similarities of these vectors, specifically, cosine similarities.
While I understand that cosine ...
1
vote
0answers
385 views
SVD item similarity calculation
I am performing SVD on a rating matrix of Users and Items and I get 3 matrices out of which Vt provides latent feature for items. How do I compute similarities between a pair of items using these ...
0
votes
1answer
217 views
How to interpret very low similarity score of two vectors but having significant permutation test
I used cosine angle to characterize the similarity of two vectors $x_{1}$ and $x_{2}$, and then performed permutation test to evaluate the significance of the similarity. For permutation test, vector $...
2
votes
3answers
724 views
Cosine similarity indexing?
Are there any open source implementations out there that can efficiently solve the following.
I'm given a fixed set $S$ of $n$-dimensional vectors of size $N$, where $N$ is of the order of a million.
...
1
vote
0answers
69 views
In what way does the RV coefficient measure similarity?
The Pearson correlation coefficient is a cosine between two vectors. That's easy to understand but what happens when instead we look at the correlation between two matrices through the RV coefficient?
...
0
votes
0answers
373 views
Why is only Euclidean distance allowed to be used for Ward's method? [duplicate]
Using scipy, I noticed that I am allowed to use only Euclidean distance for Ward's method.
Is it because Ward's uses Error Sum of Squared?
What if I use Ward's method with cosine similarity?
Cosine ...
1
vote
1answer
37 views
Measuring simmilarity of observations (non numeric)
I have a dataset of format :
day,measurement1,measurement2
1,a,b
1,a,c
1,f,s
2,a,b
2,a,c
2,f,g
3,a,d
3,a,q
3,f,s
In this example day1 is more similar with day2 ...
