Questions tagged [cosine-similarity]
An angular-type similarity coefficient between two vectors. It is like correlation, only without centering the vectors.
102
questions
0
votes
0answers
8 views
How to calculate similarity between two sets of items rated on a single dimension?
(I'm just making up variables for this example.)
Let's say I have 100 words rated on their pleasantness.
I also have 100 images rated on their pleasantness.
I then had participants rate the fit ...
0
votes
0answers
22 views
cosine of angle between random variables is equal to the correlation coefficient? [duplicate]
I have seen this said multiple times where (1) the cosine of the angle between the random variables (on a vector space) is equal to the correlation coefficient, and (2) the claim if random variables ...
1
vote
1answer
17 views
Difference between an interaction and combining two variables into one with a similarity measure?
Conceptually, what would be the difference between interacting two variables in a regression and combining these two variables into one via a similarity measure? I'm looking at top managers and boards ...
0
votes
0answers
11 views
Performance of model decreases when calculating Euclidean distance between vectors with TF.IDF weigths compared to TF weights
My problem in short:
I use Jaccard similarity, cosine similarity and euclidean distance to compute a similarity between documents. The documents consists of either the words,hashtags or combination of ...
0
votes
0answers
27 views
How to calculate the similarity of data with noise?
I'm stuck on calculating similarity. Please tell me in which direction to move.
There are three files of different lengths that need to be compared for similarity.
It is supposed to use the cosine ...
0
votes
0answers
8 views
How to manage the influence of variables for lsa cosine?
I'm building reccomendation system for movies and the following data
...
0
votes
0answers
16 views
Is a normalized cosine similarity a bregman divergence?
A Bregman divergence is defined as
$D(p,q) = F(p) - F(q) - < \nabla F(q), p-q>$
with F a strictly convex function of the Legendre type.
Squared Euclidian distance is a Bregman divergence, with $...
0
votes
0answers
10 views
How to estimate the convolutional representation of a graph from its similarity to other graph convolutional representation?
Suppose we have two graphs A and B disconnected to each other (let's say 2-hops each), within a larger graph. If the Convolutional representation of graph A is known, is it possible to estimate the ...
0
votes
0answers
19 views
Calculating similarities between two populations using embeddings
I would like to find items from population B that are most similar to an item from population A. I have the following set up:
Two sparse datasets where each row is an item (treat row index as item ID)...
0
votes
0answers
21 views
What are the conditions for 2-dimensional curve fitting when the estimator function is a type of sine and cosine?
What are the conditions for 2-dimensional curve fitting when the estimator function is a type of sine and cosine?
For example I wanted to estimate F=xy with Asin (((n*pi)/L)*x)sin ((((npi)/L)*z) ...
1
vote
0answers
27 views
Which text similarity algorithm should I use to compare the context of Instagram hashtags?
For a study I am comparing companies based on the posts written by their Instagram followers. I apply the following technique:
Nike has 1.000.000 followers.
2000 random followers of Nike are selected ...
2
votes
1answer
22 views
Should we apply PCA before calculating similarities in high-dimensional space if my observations have length 1?
I have high-dimensional space (around 20 features) and I want to calculate similarity based on the angle of observation, not the magnitude. I have a nice function that can compute euclidean distance ...
0
votes
0answers
111 views
Can a cosine similarity be high while a Pearson correlation be low for a pair of vectors?
I was reading a paper on neuro-evolution related deep learning paper.
In that paper, the authors showed the Pearson correlation between the true gradient vector and the evolutionarily estimated ...
3
votes
0answers
28 views
Normalized Levenshtein distance and triangle inequality
One question regarding to the triangle inequality of normalized Levenshtein Distance. I use the well-known form D(X,Y) = 1 - d(X,Y) / MAX(|X|,|Y|) where ...
0
votes
0answers
118 views
Right way to use Cosine Similarity in R using TermDocumentMatrix
I have a TermDocumentMatrix of ~500 documents, with ~90 terms as some kind of train set. I want to implement a Cosine Similarity function, to classify a new document. It is short documents(messages), ...
1
vote
1answer
60 views
Rao-Stirling diversity index
I need help to calculate the Rao-Stirling diversity index. I tried it several times but cannot achieve equal results to R packages yet (e.g. diverse).
I used the Rao-Stirling diversity index as ...
0
votes
0answers
21 views
Cosine Similarity for Classification to EM Cluster?
Perhaps my question sounds naive, uncovering the very little knowledge that I have in the field of Statistics, but is very urgent to get a solid answer or trigger for further insights for my concerns.
...
0
votes
1answer
26 views
How to find the list of nearest vectors if ony a vector is given?
I know there are many ways to compute similarity of two different non-zero vectors but is it possible to get a list of nearest vectors whose values are continous given a single continous vector.
Lets ...
0
votes
1answer
54 views
Cosine similarity for recommendation systems
Recently picked up recommendation systems and was going through User Based Collaborative Filtering(UB-CF).
Somewhere in the text, it specified that cosine similarity is one of the measures to find ...
0
votes
1answer
53 views
Why is the cosine similarity between these (seemingly uncorrelated) vectors so high?
I am calculating the correlations between vectors of experimental data by a variety of methods (Pearson's, cosine similarity, Euclidian distance, etc.). Most the results look fine, but occasionally ...
0
votes
1answer
66 views
Cosine similarity vs quadratic objective function
The aim is to calculate the similarity between two foods given the nutritional content of each. After some reading, it seems the most popular measure for this sort of problem is the cosine similarity ...
0
votes
1answer
97 views
Does it make sense to normalize vectors after PCA for cosine distance?
I start off with word2vec embeddings and process them in the following way:
Standardize dimensions to mean 0 and standard deviation of 1
PCA to keep the top k-dimensional eigenvector, whereby ...
0
votes
0answers
36 views
What does a high cosine similarity score mean here?
I have a set of 50,000 documents divided into two classes.
Class 1 has 5000 documents and,
Class 2 has 45,000 documents
Using word2vec embeddings, I extract 300 dimensional dense, real-valued ...
1
vote
0answers
19 views
Deep item-based recommender objective function
I'm trying to understand the following paper written by researchers at eBay that uses deep learning to overcome the problem of making recommendations when you mostly have one-of-a-kind items. A ...
0
votes
0answers
118 views
Extension of the K-Nearest neighbor algorithm to get results in different neighborhoods
I would like to use the kNN algorithm to find the closest neighbor to a vector. But I would like it to limit to a point per neighborhood (radius)
In this image, given the point in red, I would like ...
0
votes
0answers
114 views
Should I normalize the vectors by row or by column before performing cosine similarity?
I have a dataset which contains vectors of different features that generated from subtitles in movies, something like:
...
1
vote
0answers
102 views
How can I define a accuracy measure for word2vec predictions
I have a data set consisting of tags and some classes.I'm suppose to find the nearest class to each set of tags with Word2vec embeddings and cosine similarity.Each set of tags have multiple classes ...
1
vote
0answers
38 views
the accuracy of covariance between two high-dimensional vectors
Question
Is the covariance between high-dimensional vectors less accruate than covariance between two vectors in low-dimensional vecotrs?
I am asking this questio to check if there is a need for '...
0
votes
0answers
22 views
analyze time-series consine similarity matrices (in R)
I have calculated the cosine similiarity between multiple users for 10 years separately.
I can visualize and cluster the matrix of each year but I was wondering if there is a smart approach, vignette ...
2
votes
0answers
68 views
Are there advanced “cos similarity” that influence of dim size is less?
I found that the cosine similarity is
affected to the effect of "Curse of dimension" by trying the following simulation.
create(select) two vectors form uniform random numbers U[-1, 1], each dim = 2,...
0
votes
1answer
22 views
Did my text data come from two distinct distributions?
I have labeled text data from two different classes. I have calculated tfidf feature representation of all the sentences in question. I have a huge matrix where rows are sentences and columns are ...
0
votes
0answers
87 views
Set similarity as weight to ratings
I have a problem deciding which similarity function to use.
I want to find the similarity between the users based on their requirements about computer performance metrics normalized to 1. Each user ...
0
votes
0answers
30 views
Cosine similarity matrix of linearly transformed inputs
Given a matrix $\mathbf{C}$ which contains pairwise cosine similarities between rows of a matrix $\mathbf{A}$, linearly transformed by matrix $\mathbf{U}$:
$$
\mathbf{C} = K(\mathbf{UA}, \mathbf{UA})
$...
2
votes
1answer
65 views
Distance metric with characteristics of cosine and Manhattan
I'm working on a project where I want to find similarities between groups of events. So far I have expressed groups of events as vectors of event counts and computing similarities between them. I'm ...
1
vote
1answer
511 views
Finding similar text - algorithms and evaluation
I've been asked to create a program that will rank similar texts to an input text given a collection of text.
So far I've been using a tdidf representation and cosine similarity with a lot of regex-...
1
vote
1answer
260 views
Calculating similarity between two lists: high cosine similarity, but high RMSE
I want to see how similar two datasets are, as a way to justify that they can be used in similar contexts. In practice one dataset contains manually calculated data, and the other automatically ...
4
votes
1answer
2k views
Weighted Cosine Similarity
To convert cosine similarity to weighted cosine similarity, one can use at least two approaches. But I don't know which one is better.
The first approach is to first reweight each vector and then ...
1
vote
1answer
213 views
Spectral Clustering of a skipgram model
I have a model where I'm applying Spectral Clustering to frequencies of words. My pipeline consists in TF-IDF, followed by a <...
0
votes
1answer
169 views
Text Similarity - Cosine - Control. Suggestion to another / better method?
I would like to ask you, if anybody could check my code, because it was behaving weird - not working, giving me errors to suddenly working without changing anything - the code will be at the bottom.
...
0
votes
1answer
45 views
Similarity index between two texts Ask Question
I'm trying to compare two vectors in a small NLP project using Python. Code doesn't make any difference since I'm using scikit-learn, but my doubts are about my calculations.
I have a query vector ...
2
votes
2answers
294 views
How to fit laplace/exponential distribution to cosine similarities?
I am a computational biologist with little experience fitting data.
I'm trying to fit a distribution of cosine similarities computed between sparse matrices. The goal is to be able use this ...
0
votes
0answers
73 views
How to create a binary threshold from Cosine distance between 1-D arrays?
I have a graph of the Cosine distance between the question and the sentence most similar to it when there is an answer and when there is none.
I want to establish a threshold on the abscissa axis ...
1
vote
2answers
814 views
Cosine-Similarity vs non-linear measures
In NLP, people often use cosine similarity to measure how close two vector spaces are to each other. However, we know that cosine-similarity is the same thing as Pearson correlation, for centered ...
0
votes
1answer
168 views
K-means for data sets with scalar and vector objects
My question consists of two parts, both possibly closely related:
Part 1:
I have a dataset where the incoming data ($x$) will be an eigenvector ($V$) and an associated eigenvalue ($\lambda$). That ...
9
votes
3answers
7k views
Curse of dimensionality- does cosine similarity work better and if so, why? [duplicate]
When working with high dimensional data, it is almost useless to compare data points using euclidean distance - this is the curse of dimensionality.
However, I have read that using different distance ...
0
votes
1answer
129 views
Correctness of a skewed cosine similarity graph
I am currently implementing a word2vec model that uses the cosine similarity to determine the similarity between two vectors.
When plotting all the possible cosine similarities, I get the following ...
0
votes
2answers
3k views
How to find nearest neighbors using cosine similarity for all items from a large embeddings matrix?
I have an embeddings matrix of a large no:of items - of around 100k, with each embedding vector length of 100. So a matrix of size 100k x 100;
From this, I am trying to get the nearest neighbors for ...
1
vote
0answers
532 views
User-user suggestions with collaborative filtering (recommendation system)
I have a binary matrix N x N where both rows and columns represent users of a website. If matix[i,j] = 1 it means that ...
0
votes
1answer
235 views
Word vector normalisation by document size
I have a bunch of text documents of varying lengths (100k words to just thousands). I want to compare similarities of these vectors, specifically, cosine similarities.
While I understand that cosine ...
2
votes
0answers
479 views
SVD item similarity calculation
I am performing SVD on a rating matrix of Users and Items and I get 3 matrices out of which Vt provides latent feature for items. How do I compute similarities between a pair of items using these ...
