Questions tagged [distance]
Measure of distance between distributions or variables, such as Euclidean distance between points in n-space.
571
questions
0
votes
0answers
9 views
Distance metric for probability distribution
I'm looking for a distance metric that would be good for what I think is a probability distribution:
I have a fixed number of samples
I have a fixed number of features
Each sample can have, at most,...
1
vote
0answers
7 views
How to calculate similarities/distances and conduct cluster analysis from contingency table
I have an 18x25 contingency matrix, rows are behaviors observed and columns are environmental features. I conducted a correspondence analysis via the CA() function in R and then clustered using the ...
0
votes
0answers
13 views
Creating a fraud score for an image fraud detection workflow
So to set the context. I'm working on an image fraud detection workflow for an insurance company. The idea basically is:
There are a number of cases (car accidents).
Each case contains a set of ...
4
votes
1answer
24 views
What happens if i cluster data with a distance metric, that is not a distance metric?
i stumbled upon a paper, that introduces a distance metric, which is then used to cluster data (https://doi.org/10.1137/1.9781611972795.35). I noticed however, that this "distance" violated at least ...
3
votes
0answers
32 views
Are there statistical methods for determining whether a 2D distribution is “inside” a region in the cartesian plane?
So I know that t-tests and ANOVAs are used to determine if the means of 2 normally distributed random variables are significantly different, and the p-values give a statistical confidence level of ...
0
votes
0answers
12 views
Compare differences between 2 datasets with Jensen-Shannon divergence
I have two medical imaging datasets:
Brain MRI from institution A
Brain MRI from institution B
Because of different acquisition parameters the intensity distribution in those datasets could be ...
0
votes
0answers
12 views
Kruskal's Stress for MDS: How to compute this in R?
I am performing classical MDS on a dataset (Gower matrix returned by R "Cluster" package function daisy). In my field, a measure of fit of the MDS is reported. Other researchers usually perform this ...
0
votes
0answers
18 views
Goodness of fit that puts high weight towards the tail of the distributions
I have two distributions A and B and I am looking for a goodness of fit test that measures how much the tail of A matches (or fail to match) the tail of B. Alternatively, I am looking for a test that ...
0
votes
0answers
7 views
Distance Metric for Sparse Data
I'm aware of similar questions like Distance metric for categorical and numerical data. I have a very sparse matrix and I want to a distance metric to find the most different items. Would performing ...
0
votes
0answers
11 views
What is the difference between np.linalg.norm(x-y,axis=1) and np.linalg.norm(x-y)?
I'm creating a K-Medoids algorithm from scratch in Python using numpy, and I'm in the process of using a distance function to determine the cluster center. I want the center to be the point in the ...
0
votes
1answer
26 views
What output from PCA do I use for a Mahalanobis distance analysis?
I am working with dental metric data to perform a biological distance analysis. The standard procedure in my field is to perform a PCA on the cleaned, imputed dataset to reduce correlation and then ...
0
votes
0answers
10 views
Is a normalized cosine similarity a bregman divergence?
A Bregman divergence is defined as
$D(p,q) = F(p) - F(q) - < \nabla F(q), p-q>$
with F a strictly convex function of the Legendre type.
Squared Euclidian distance is a Bregman divergence, with $...
0
votes
0answers
8 views
How to mathematically prove that cost function decrease if the ācentroidā is updated in K means?
How to mathematically prove that cost function decrease if the ācentroidā is updated in K means?The cost function is :
This cost function ,which is the sum of square of the distances of each point to ...
0
votes
1answer
21 views
What happened when k=1 in k means? What's the optimized value of distance for k=1?
What is the optimized value of distance V(x,c) when k=1 (number of cluster) in k means? What is the centroid such that it is optimal?
which is the sum of square of the distances of each point to the ...
0
votes
0answers
25 views
Interpreting exercise in Elements of Statistical Learning
I am reading exercise 6.4 from The Elements of Statistical Learning (Hastie, Tibshirani and Friedman) and I am having difficulty interpreting exactly what is being asked in the following question
...
1
vote
2answers
47 views
Measuring the distance between two probability measures using quantile functions?
There are many metrics on the space of probability measures on $(\mathbb R, \mathcal B)$.
Most of the famous metrics use the distribution functions associated with the probability measures to ...
0
votes
0answers
7 views
Covariance matrix distance variability
I have been preparing an analysis of a data set from which I have extracted a number of covariance matrices and used the Log-Euclidean algorithms to calculate distances, geodesics etc. This data set ...
0
votes
0answers
18 views
Is the KL-Divergence invariant to strictly monotonic transformations of the random variable?
Let $p$ and $q$ be two distributions on a variable $X$. Let $\widetilde{p}$ and $\widetilde{q}$ be the corresponding distributions on $f(X)$, where $f$ is a strictly monotonic function (e.g. $f(x)=e^x$...
0
votes
0answers
6 views
Bounding the distributional error introduced by bootstrapping
Suppose I have some data $x_N$ of `size' $N$ which were drawn according to some measure $P_N(x_N)$.
I'm imagining $x_N$ as being any of:
an exchangeable / iid sequence of length $N$
a stationary ...
0
votes
0answers
8 views
How best to compare Euclidean distance error and latency across two different scales?
I have two sets of data from two different behavioural tasks, in which participants make judgements as to where an object should be within a circle. In one task, the circle has a diameter of 6 metres, ...
0
votes
1answer
29 views
What is the right way to test significant for paired distance matrices?
I have distance matrices of two clusters, each one represents the genomic distance between all of the cluster's species.
Here is a density plot of the distance values within the clusters between the ...
0
votes
1answer
28 views
How to get the variance between two arrays?
import numpy as np
a = np.array([[1,2,4],[2,4,8]])
np.var(a)
output: 5.25
Can anyone enlight me what's the calculation process to get variance = 5.25?
1
vote
1answer
72 views
Intuition behind Weight of Evidence and Information Value formula
In credit scoring models, we use Weight of Evidence to create bins for continuous variables and Information value to filter out important variables.
\begin{align}
\text{WoE:} \qquad &\ln \frac{\...
0
votes
0answers
5 views
Relationship between Principal Component Analysis and Multidimensional Scaling? [duplicate]
I understand the concept of MDS, but I am struggling to understand the similarities between the two.
2
votes
0answers
71 views
Distance metric between structured 2D scenes
I am working on a problems that consist of simple 2D scenes, a typical example being a top-down traffic situation encountered by an autonomous vehicle. These structured scenes have multiple actors ...
2
votes
1answer
32 views
Earth mover's distance implementation for circular distributions?
I am interested in finding an implementation (preferably in R but not necessary) to calculate earth mover's distance between two empirical distributions of circular data. My data are time points on a ...
0
votes
0answers
18 views
Calculating similarities between two populations using embeddings
I would like to find items from population B that are most similar to an item from population A. I have the following set up:
Two sparse datasets where each row is an item (treat row index as item ID)...
0
votes
1answer
23 views
Are Bhattacharyya coefficient and total variation distance complementary?
I was reading about total variation distance, and, as I understood it, it should measure how much two probability measures don't overlap. To be clear: in these images Bhattacharyya coefficient is ...
5
votes
1answer
69 views
Best practices in the selection of distance metric and clustering methods for gene expression data
I have been reading about this on various channels including here and Stack Exchange, but I'm still not sure how to choose the best approach for clustering gene expression data. As a Ph.D. molecular ...
0
votes
2answers
37 views
How to emphasize a sudden drop in time series for the purpose of clustering?
I would like to cluster uni-variate daily time series so that an emphasis is put on sudden drops in time series. Series that contain such uncommon drops should be in one cluster (drops should ...
0
votes
0answers
5 views
How to identify the dissimilar points between multiple time series having almost similar patterns?
I have multiple time series that are quite similar to each other in terms of pattern. I Clustered all them to get similar time series under a cluster.
This is what the cluster looks like:
Now I'd ...
0
votes
0answers
11 views
Similarity between classes after Random forest classification
Naive question, is it possible to extract the distance (or how similar the classes are to each other) between the classes after Random forest?
I classified 15 different classes which contains ...
1
vote
1answer
40 views
Kendall distance significance testing
Recall, Kendall distance is a manipulation of Kendall tau, in that it only considers discordant pairs, and therefore is a dissimilarity measure. It ranges from 0-1 and is calculated as follows:
$d= \...
0
votes
0answers
7 views
Comparing Parametric Distributions with Empirical Parameters
Suppose I have two data sets, $D_1$ and $D_2$, and a parametric distribution with parameter $\theta$. Using some standard method (i.e., MLE), I can certainly fit $\hat\theta_1$ and $\hat \theta_2$ to ...
0
votes
0answers
13 views
Suitable similarity measure for zero-rich preference data
I am performing some statistical analysis on preference data (-10 for strong dislike, 0 for neutral, 10 for strong like, let's assume all floats in [-10,10] are allowed) for 30 different food items ...
1
vote
0answers
22 views
Which text similarity algorithm should I use to compare the context of Instagram hashtags?
For a study I am comparing companies based on the posts written by their Instagram followers. I apply the following technique:
Nike has 1.000.000 followers.
2000 random followers of Nike are selected ...
0
votes
0answers
16 views
L1 version of Pearson correlation?
Instead of Euclidean distance I typically use Manhattan distance. This is because Manhattan distance does not give enormous weight to outliers the way Euclidean distance does.
Is there a Manhattan-...
0
votes
0answers
20 views
Does DTW return smaller distance measure than Euclidean Distance?
QUESTION 1: When computing the distance between two time series, shouldn't the DTW distance measure return a smaller distance than the Euclidean distance (assuming DTW internally uses the Euclidean ...
0
votes
0answers
19 views
Finding weights to be used by an Euclidean distance function for vectors with weighted components in a multi-dimensional space, using only sample data
I have a multi-dimensional space with $d$ dimensions wherein $i$ vectors ($v_1 ... v_i$) with $d$ components live. I want to find a function $s(v_a, v_b)$ which takes in two of these vectors and ...
0
votes
0answers
31 views
relation between Gromov Wasserstein distance and p Wasserstein distance
I am trying to understand the relationship between the p-Wasserstein distance implemented here :
https://pot.readthedocs.io/en/stable/all.html#ot.wasserstein_1d
and the Gromov Wasserstein distance: ...
0
votes
1answer
113 views
Combining two distance matrices for clustering
I'm looking to see if I can cluster geographic regions by a variable, such as losses.
For example, I have 5 regions with the following amount of losses:
Region 1 with \$500,000 in losses.
Region 2 ...
1
vote
1answer
28 views
Are the conditions of metric space satisfied in the latent space of a classification task?
Specifically, in the case of a neural network trained in a categorical classification task (cross-entropy loss function), does the final layer embedding space preserve the definition of distance ...
0
votes
0answers
10 views
Finding the relative 'importance' of different vector components when defining distance of two vectors in a space [duplicate]
I have a multi-dimensional space with $d$ dimensions wherein $i$ vectors ($v_1 ... v_i$) with $d$ components live. I want to find a function $s(v_a, v_b)$ which takes in two of these vectors and ...
0
votes
0answers
11 views
Finding distance/similarity function between two vectors in a multi-dimensional space, with weighted components/dimensions [duplicate]
I have a multi-dimensional space with $d$ dimensions wherein $i$ vectors ($v_1 ... v_i$) with $d$ components live. I want to find a function $s(v_a, v_b)$ which takes in two of these vectors and ...
1
vote
1answer
56 views
Best path through the matrix in dynamic time wrapping: interpreting and implementation
I read several papers and blog to understand how dynamic time wrapping (DTW) can be used to compare two time series data. I understand how to generate matrix and also understand how to choose best ...
3
votes
1answer
38 views
Upper Bound on the Wasserstein Distance
I'm interested to know if it's possible to construct an upper bound on the Wasserstein distance in terms of the Kolgomorov distance.
The Wasserstein distance can we written as
$$W_{1}\left(F, G\...
0
votes
0answers
8 views
Can I calculate distances between categorical variables with domain specific knowledge?
I have a dataset of beers containing (Percentage, color, taste, what kind of beer and what kind of fermentation).
Color taste and the 2 kinds are categorical variables. And they are not really ...
0
votes
0answers
32 views
Kullback-Leibler (KL) divergence cutoff value
I am performing the KL divergence method to compare distributions of variables between two groups. I have a list of variables within different categories (award types, organization types, topics, etc) ...
0
votes
0answers
25 views
Weighted pairwise distance on categorical data
I have data on concentration values for each sample. These data are ordinal, not continuous, as they have a specific set of possible values, like this:
...
1
vote
0answers
323 views
How to use Gower's Distance with DBSCAN algorithm in Python
I have been researching about using DBSCAN with sklearn in python but it doesn't have Gower's distance metric built in. All the other implementations are in R in this community.
I'm using a dataset ...
