Skip to main content

Questions tagged [cosine-similarity]

An angular-type similarity coefficient between two vectors. It is like correlation, only without centering the vectors.

Filter by
Sorted by
Tagged with
0 votes
0 answers
9 views

Similarity measure for sparse, ordered, binary vectors, with more weighting to True values

I have two sparse, ordered, binary vectors. The size of the vectors is around ~100. I am under the impression that cosine similarity is useful for sparse, ordered, binary vectors. For my purposes, it ...
feonyte's user avatar
  • 69
0 votes
0 answers
10 views

Appropriateness of the Universal Sentence Encoder model

I have a classification problem where the goal is to predict, based on a small paragraph, if an individual is British or not. The model used for the classification is Universal Sentence Encoder (to ...
Sara Mun's user avatar
0 votes
0 answers
99 views

Locality sensitive hashing (LSH) with word embeddings and cosine similarity

I would like to ask about the methodology of LSH algorithm with Word Embeddings and Cosine Similarity to identify similar documents. First, I tokenize my sentences to create a list of tokens. Then, I ...
BDEngineer's user avatar
0 votes
2 answers
204 views

A Simple Toy ML problem that surprisingly fails to converge (or even "try"!)

This is a much simplified network from a real problem that, to me, has a surprising INability to learn a simple task via backprop, ie, it can't overfit or learn at all. This simple version has come at ...
Josh.F's user avatar
  • 107
0 votes
0 answers
58 views

Calculate the cosine similarity between two probability measures, rather then between two vectors

Introduction. The cosine similarity is often calculated among two vectors, i.e. $C_S(X,Y)$: \begin{equation} C_S(X,Y) = \frac{\sum_{i=1}^{n}(X_i,Y_i)}{\sqrt{\sum_{i=1}^{n}X_{i}^{2} \sum_{i=1}^{n}Y_{i}^...
Ommo's user avatar
  • 280
0 votes
0 answers
21 views

How to adjust similarity scores by removing the influence of a common vector?

I have a similarity score function, $s(x,y)$. I know that I have two items that I'm trying to compare the similarity of, but both are based on the same template. How would I remove the template from ...
user760900's user avatar
0 votes
0 answers
39 views

Similarity measure for non-binary user preference vector

Cosine similarity can be used to measure the similarity between two vectors that encode the user preferences with the following values: 0: No user feedback (e.g. the user never viewed a merchandise) ...
Zizheng Tai's user avatar
2 votes
1 answer
303 views

Normalizing Euclidean distance by the length of the vectors [closed]

Suppose I have 4 vectors, the first 2 vectors are of length 4 and the last 2 vectors are of length 400. all values in the vectors range from 0.5 to 0.6. The Euclidean distance between the last 2 ...
user17420392's user avatar
1 vote
1 answer
180 views

Is it wrong to view convolution as template matching?

I am reading about the convolution operation but I can't see how it can be seen as template matching. Suppose that we convolve the input $\mathbf{X}$: $$ \begin{bmatrix} 1 & 0 & 0 & 0 \\ 0 ...
ado sar's user avatar
  • 477
2 votes
1 answer
44 views

Composed cosine similarity

I have the following problem. I have 3 vectors $u,v,w$ of n dimensions. I'm able to find cosine similarities between $u$ and $v$, and between $v$ and $w$: $cosine(u,v)$ and $cosine(v,w)$. Can i use ...
Chg's user avatar
  • 21
0 votes
1 answer
148 views

How to interpret NearestNeighbor results obtained using cosine similarity for tf-idf vectors

Why is the top result obtained using cosine similarity extremely close to 0 not the expected 1? That implies complete orthogonality. Data: 100k documents/rows with 2000 features(TF_IDF values of ...
Sint's user avatar
  • 111
0 votes
1 answer
150 views

Can you calculate the mean of a multivariate normal distribution for the cosine similarity?

Let ${\displaystyle \mathbf {X} =(X_{1},\ldots ,X_{k})^{\mathrm {T} }}$ be a sample from a multivariate normal distribution ${\displaystyle \mathbf {X} \ \sim \ {\mathcal {N}}({\boldsymbol {\mu }},\,{...
lupus83's user avatar
0 votes
0 answers
26 views

What’s a good scale robust similarity metric?

Context: While designing an experiment, there is interest from leadership in pairing control and test units together such that every (test, control) pair are as categorically similar as possible. ...
jbuddy_13's user avatar
  • 3,394
1 vote
0 answers
22 views

Issue on computing correlation coefficient for two vectors

This is a super trivial question, but for some reason I am stuck and I simply don't get what I am missing/doing wrong. The cosine similarity is defined as followed: I have this dataset, of points ...
Anon's user avatar
  • 11
2 votes
0 answers
352 views

What does the PCA similarity factor tell us?

I found the following formula for comparing time series $$ S_{\mathrm{PCA}}^{\lambda}=\frac{\sum \limits_{i=1}^{k} \sum \limits_{j=1}^{k}\left(\lambda_{i}^{(1)} \lambda_{j}^{(2)}\right) \cos ^{2} \...
Unistack's user avatar
0 votes
0 answers
21 views

Should I do documents transformation at once or a pair at a time for auto grading with cosine similarity?

I'm developing auto grading essay that compares the similarity between the answer key and student answer with cosine similarity. This one is written in php. Let's say in a course there are 30 - 100 ...
newtocoding's user avatar
1 vote
0 answers
51 views

Similiarity between two corpus of text

I have two separated corpus of text, and i would like to understand wheter these are similiar or not using cosine similarity. I'm not sure on how to approach this problem, but i was thinking as a ...
user373562's user avatar
1 vote
1 answer
114 views

Geometric Explanation of R-Squared

Suppose we have two orthogonal features $x_1$ and $x_2$. If we run univariate regression of $y$ on $x_1$ or $x_2$ and get $R^2, $$r_1$ and $r_2$ and we run multivariate regression of $y$ on both $x_1$ ...
DA_PA's user avatar
  • 397
1 vote
1 answer
28 views

Algorithm to find closest document containing a set of strings, or variations of them

I have one dataset (A) containing several fields (strings) per sample. One of these fields is a name, and the others are all alphanumeric identifiers. I have another dataset (B) which contains highly ...
KOB's user avatar
  • 465
3 votes
1 answer
350 views

Centering in normalized cross correlation for template matching

Context I'm following Lewis (1995) exposition on normalized cross correlation for template matching (Section 2). The cross-correlation of the image and the feature at $u,v$ is denoted by $c(u,v)$ and ...
Simón Ramírez Amaya's user avatar
0 votes
0 answers
151 views

Is cosine similarity enough to measure word embedding similarity?

Is cosine similarity a good metric to measure word embedding similarity? Suppose that we have two vectors of word embedding in same direction but with different length( first one with len=1 and second ...
Mahdi Amrollahi's user avatar
1 vote
0 answers
732 views

Cosine Similarity of the word embeddings after UMAP dimensionality reduction

I want to calculate similarities of the word embeddings. As a basis I took SpaCy german corpus: nlp = spacy.load("de_core_news_lg"). I have approximately ...
Daniil Yefimov's user avatar
1 vote
0 answers
255 views

How to calculate euclidian distance from similarity matrix

I have a similarity matrix but can not use it in the k-mean as input. So is there any way to convert the similarity matrix to the euclidean distance matrix? Edit: I have Jaccard, overlap, cosine,(...
Segmentation fault's user avatar
0 votes
0 answers
471 views

Scalar similarity measure between two vectors including both angle and magnitude

I have different models, predicting a vector $\boldsymbol{v}\in\mathbb{R}^3$. Now I would like to compare the performance of these models against a baseline vector $\boldsymbol{b}\in\mathbb{R}^3$, for ...
reox's user avatar
  • 261
2 votes
1 answer
2k views

Cosine similarity seems to perform better with higher dimensions than Euclidean distance? Should this be the case?

I've generated 100 random vectors (data points) in n∈[1,...,50] dimensions. I then compared distances between each pair of vectors and calculated the mean value. I've done this for all dimensions ...
Jamess11's user avatar
0 votes
0 answers
84 views

How Can I Reduce Similarity Analysis of Multiple Time-Series Vectors into a Single Value?

I have ~15 independently-sourced vectors with about 1600 samples in each. They are basically continuous, ~1 Hz, from t=0 to 22 minutes. The nature of the dataset is such that the signals are ...
Clayton Schneider's user avatar
0 votes
1 answer
79 views

Search, rank and recommend in large text datasets

Imagine you are Spotify and you have billions of songs. Assume that each of these songs are transcribed into text. How do you design your search and recommendation pipeline such that when somebody ...
mhsnk's user avatar
  • 307
0 votes
0 answers
87 views

Why is cosine similarity a lot higher between one document and all other documents grouped than vs the mean of each

I am trying to calculate a cosine similarity score for speeches made by various politicians. Say I have politician A who belongs to party A. Party A has 20 other politicians who have made a total of ...
Andy's user avatar
  • 1
1 vote
1 answer
970 views

cosine similarity with weights

We're doing pairwise similarity computation for some real estate properties. Our data goes something like this: ...
kms's user avatar
  • 560
0 votes
0 answers
49 views

measure the overall similarity between two vectors

I have two paired vectors which report the number of times each object was found in the two experiments (the count value could be 0 or greater (e.g. 5,10,1000, etc)). I want to compare them and get a ...
pingu87's user avatar
  • 21
1 vote
0 answers
386 views

Cosine similarity with vectors of different lengths (time series)

I have time series of different lengths with different patterns. I want to find the similarity between them in such a way that it is defined from 0 to 1 and the measure takes into account the length (...
SoH's user avatar
  • 33
0 votes
0 answers
60 views

Searching for a distance metric or distance similarity to compare two arrays of counts

Every object in my dataset is described as a vector of n = 200 continuous counts. I need to choose a measure to evaluate the similarity of two objects in the dataset. I want to satisfy the following ...
cag104's user avatar
  • 3
1 vote
0 answers
49 views

Should I consider the contribution of a variable in correspondance analysis if the $cos^2$ is weak?

So I have this table that represents contribution and cosinus of variable in CA I noticed that the most popular choice ( 1-5 hours ) has the weakest contributions and the weakest cosinus. It made ...
wageeh's user avatar
  • 241
1 vote
0 answers
105 views

Cosine similarity with recentering - collaborative filtering

I don’t know much about stats (nor maths), so I’m sorry if I’m not being very clear on this... I’m trying to build a simple recommender system for books using collaborative filtering item by item. I’...
OneMinute's user avatar
2 votes
0 answers
285 views

Question related to using Pooled Output from BERT for similarity between sentences

I was hoping someone could give me advice and feedback on my current approach and possibly suggest to me a possible alternative. I'm trying to find the sentences that are most similar using the pooled ...
marco's user avatar
  • 21
3 votes
0 answers
981 views

Finding Cosine similarity threshold (cut-off point) with Logistic Regression

I want to preprocess my data sets, removing similar articles. To do this, I'll perform cosine similarity, I don't know the best threshold (cut-off point) of my data set. I Found in this paper, there's ...
Cookie Monster's user avatar
0 votes
0 answers
922 views

Calculate similarity between two matrices

I have two matrices, $A$ and $B$, each of size $n\times m$, where $n$ is discrete time points, and $m$ are the variables measured (specifically, $n$ are dates and $m$ are investments measured in ...
Arthur D.'s user avatar
  • 233
1 vote
1 answer
754 views

Baffled by cosine similarity - these results seem counterintuitive

Haven't used cosine similarity much in the past so getting into it now. Seeing results that are counterintuitive and would love your help making sense of them. Assume these simple vectors: ...
Optimesh's user avatar
  • 173
1 vote
0 answers
618 views

Euclidian distance vs cosine similarity

Currently I'm working on facial recognition. If I use encoding/feature vectors of 2 images which method will prove more accuracy, L2 norm or cosine similarity and why? I read "ICA performs ...
offset-null1's user avatar
2 votes
1 answer
758 views

Cosine similarity for Categorical datasets?

Can I use Cosine similarity measure for estimating similarity/relationship between D1 and D2 (two categorical datasets)
Msilvy's user avatar
  • 51
0 votes
0 answers
48 views

How to calculate similarity between two sets of items rated on a single dimension?

(I'm just making up variables for this example.) Let's say I have 100 words rated on their pleasantness. I also have 100 images rated on their pleasantness. I then had participants rate the fit ...
Dave's user avatar
  • 2,651
1 vote
0 answers
33 views

cosine of angle between random variables is equal to the correlation coefficient? [duplicate]

I have seen this said multiple times where (1) the cosine of the angle between the random variables (on a vector space) is equal to the correlation coefficient, and (2) the claim if random variables ...
student010101's user avatar
2 votes
1 answer
838 views

Difference between an interaction and combining two variables into one with a similarity measure?

Conceptually, what would be the difference between interacting two variables in a regression and combining these two variables into one via a similarity measure? I'm looking at top managers and boards ...
Charlie Glez's user avatar
0 votes
0 answers
44 views

What are the conditions for 2-dimensional curve fitting when the estimator function is a type of sine and cosine?

What are the conditions for 2-dimensional curve fitting when the estimator function is a type of sine and cosine? For example I wanted to estimate F=xy with Asin (((n*pi)/L)*x)sin ((((npi)/L)*z) ...
Beh's user avatar
  • 1
1 vote
0 answers
180 views

Which text similarity algorithm should I use to compare the context of Instagram hashtags?

For a study I am comparing companies based on the posts written by their Instagram followers. I apply the following technique: Nike has 1.000.000 followers. 2000 random followers of Nike are selected ...
nielsvg's user avatar
  • 21
3 votes
1 answer
1k views

Should we apply PCA before calculating similarities in high-dimensional space if my observations have length 1?

I have high-dimensional space (around 20 features) and I want to calculate similarity based on the angle of observation, not the magnitude. I have a nice function that can compute euclidean distance ...
jakes's user avatar
  • 215
3 votes
0 answers
292 views

Normalized Levenshtein distance and triangle inequality

One question regarding to the triangle inequality of normalized Levenshtein Distance. I use the well-known form D(X,Y) = 1 - d(X,Y) / MAX(|X|,|Y|) where ...
Yukun's user avatar
  • 31
1 vote
1 answer
390 views

Rao-Stirling diversity index

I need help to calculate the Rao-Stirling diversity index. I tried it several times but cannot achieve equal results to R packages yet (e.g. diverse). I used the Rao-Stirling diversity index as ...
KGM's user avatar
  • 11
1 vote
1 answer
241 views

How to find the list of nearest vectors if ony a vector is given?

I know there are many ways to compute similarity of two different non-zero vectors but is it possible to get a list of nearest vectors whose values are continous given a single continous vector. Lets ...
Eka's user avatar
  • 2,281
1 vote
1 answer
866 views

Cosine similarity for recommendation systems

Recently picked up recommendation systems and was going through User Based Collaborative Filtering(UB-CF). Somewhere in the text, it specified that cosine similarity is one of the measures to find ...
user1673216's user avatar