Questions tagged [distance]
Measure of distance between distributions or variables, such as Euclidean distance between points in n-space.
736
questions
0
votes
0
answers
36
views
In a principal component space, is a change in any coordinate going to result in the same distance between PC-projected observations?
Suppose that we have observations in a 2-dimensional principal coordinate space. Let's denote one observation $\mathbf{y}=(y_1,y_2)$ in our PC space. Further, suppose that we have another observation $...
- 2,129
1
vote
0
answers
70
views
Measuring distances between distributions of ordinal variables
I'd like to be able to measure how "different" two distributions of ordinal (but not interval) variables $X$ and $Y$ are. Given three random variables $X$, $Y$, $Z$ I'd also like to be able ...
- 31
1
vote
0
answers
75
views
R adjusted cosine similarity [closed]
I would like to find an effective way to get adjusted cosine similarity for a large matrix (10k rows).
Apply seems also slow...
Could you please suggest a method?
Maybe based on the approaches ...
- 11
1
vote
0
answers
283
views
What are the downsides of using euclidean distance for hierarchical clustering of a correlation matrix?
Apologies if this has been answered elsewhere, but I couldn't find any answers discussing this specific question.
I am lacking some notion on clustering using euclidean vs correlation distance, when ...
- 11
2
votes
0
answers
69
views
How to calculate the max possible earth mover distance between histograms, given the buckets?
I have two histograms and am able to calculate the EMD / Wasserstein metric between them using the algorithm described here. In order to better communicate the implication of this metric, I want to ...
- 71
1
vote
1
answer
838
views
Earth Mover (Wasserstein) distance for ordinal discrete data
I am doing data analysis for my Masters research and which includes some Likert scale type questions. I have been calculating some distances between the responses for these questions. All this has ...
- 121
4
votes
1
answer
206
views
Why do we not have a true upper limit for dissimilarity measure?
The similarity measure between two attributes always falls in the range of $[0,1]$, why is this not true in the case of dissimilarity where the value falls in the range of $[0, \infty)$?
Can't the ...
0
votes
0
answers
87
views
Euclidean distance between points in high dimensions
On Wikipedia there's a statement:
When a measure such as a Euclidean distance is defined using many coordinates, there is little difference in the distances between different pairs of samples.
Is ...
- 1,169
2
votes
1
answer
79
views
In DBSCAN, what happens if points have distance exactly equal to the Epsilon radius of a core point?
In DBSCAN the border points are points in the eps-neighborhood of a core point. But what if a point has distance exactly equal to Epsilon from a core point? Is it considered inside the eps radius, or ...
- 43
1
vote
1
answer
148
views
How do I compare multivariate normal distributions and get a p-value?
I have sample-data for two multivariate normal distributions. From this sample-data, I can calculate each distribution’s parameters (means and standard deviations).
How do I quantify the distance (or ...
- 13
2
votes
0
answers
69
views
A random perturbation of a distance matrix
Let $A=(d_{ij})$ be a distance matrix, i.e.,
$d_{ii}=0$ for all $i$
$d_{ij}>0$ for all $i\neq j$
$d_{ij}=d_{ji}$ for all $i,j$
$d_{ij}+d_{jk}\geq d_{ik}$ for all $i,j,k$
How to find a random ...
- 321
1
vote
0
answers
36
views
Expected value of inverse distance between two 3D normal distributions
Consider two independent trivariate normal random variables $X$ and $Y$. The means are non-zero and the off-diagonal elements of the covariance matrices are non-zero. $X$ and $Y$ does not follow the ...
- 216
0
votes
2
answers
750
views
Statistical difference between curves
I have 12 curves (three replicates for each treatment), see attached picture. X=days; Y=percentage.
The experiment has 2 variables: A: 4 cases; B: 3 cases.
I would like to know, if there is a method ...
- 31
0
votes
1
answer
124
views
Can we test if distance matrices are significantly farther apart?
I work in the field of linguistics, and my current project involves lots of distance matrices generated from language data, which measure the distance and similarity of dialects. Concretely, my ...
2
votes
2
answers
178
views
Calculating distances: Using count data or discrete probabilities?
I am looking into calculating distances between vectors for some data analysis. One question I have is whether I should use actual count data or convert to discrete probabilities.
For some distances, ...
- 121
1
vote
1
answer
185
views
Distance measure between two sequences of different lengths [duplicate]
I am looking for some distance (or similarity) measure between two sequences, possibly of different lengths. Conceptually, I would like a measure with a property that $[3,1,5]$ is similar to $[3,1,4,5]...
- 554
2
votes
1
answer
68
views
Is this Hellinger Distance expression correct?
Further to my previous post on the Hellinger Distance, there was one comment raised about there being different expressions of the Hellinger Distance. This has intrigued me.
In the Encyclopedia of ...
- 121
4
votes
1
answer
110
views
Equating different forms of Hellinger Distance
For a research report, I want to show that the standard expression of the Hellinger Distance between two discrete distributions,
$$H(p,q)={\sqrt{\frac{1}{2} \sum_{x \in X} \left[\sqrt{p(x)}-\sqrt{q(x)}...
- 121
1
vote
2
answers
86
views
What type of statistical test can I use to look at the influence of a factor on a frequency distribution (distance)
I'm trying to work out what test to use to see whether a factor (presence vs absence) affects a frequency distribution with distance as a predictor.
Just wondering if anyone knows what the best test ...
- 31
2
votes
0
answers
31
views
Distance metric for vector of functions and real numbers
I want to cluster points which have the following feature vector:
$$\theta = (f_1, f_2, \cdots, f_n, y_1,y_2,\cdots,y_m)$$
i.e. the first $n$ entries are functions $f_i$ ($\mathbb{R}\rightarrow \...
- 235
0
votes
1
answer
687
views
Distribution of distances between points in Gaussian sample
Given a sample of $N$ points $\{\mathbf{x}_i\}$ from a bivariate Normal distribution (arbitrary mean and covariance matrix, though I'm mainly interested in the zero-mean case), how does one derive the ...
- 101
2
votes
1
answer
214
views
For the logit model, how is the maximum likelihood and minimum distance estimator related?
Suppose I have data $\{y_i,x_i\}_{i=1}^N$, where $x_i\in\{s_1,...,s_K\}$ and follows a discrete uniform distribution. For each realized $x_i$, $y_i$ is generated by the logit model, i.e., $Pr(Y_i=1|...
- 2,778
2
votes
2
answers
590
views
Comparing PCAs using the distances between the centroids of groups
For example, I have a data for N countries where quantitative values for men and women are given. I ran PCA separately for the data from each country. PC1 and PC2 explain most of the variance, so I ...
- 21
1
vote
1
answer
22
views
Algorithm to find closest document containing a set of strings, or variations of them
I have one dataset (A) containing several fields (strings) per sample. One of these fields is a name, and the others are all alphanumeric identifiers.
I have another dataset (B) which contains highly ...
- 385
0
votes
0
answers
19
views
Transform Earth mover's distance/Wasserstein metric to percentance (0 to 1 or 0% to 100%) [duplicate]
Ok, I do some calculations between binned histograms via the use of Wasserstein metric and more specifically this python library.. https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats....
- 157
0
votes
0
answers
30
views
Hybrid correlation/MSE metric?
Apologies as this is probably a rather naive question
I would like to compare two vectors and neither a correlation (e.g pearsons/spearmans) nor a distance metric (eg rmse) is quite right and ideally ...
- 253
0
votes
0
answers
16
views
Why $P$ and $Q$ don't exist on the same coordinate, they need to be reconciled (processed) to exist on the exact same cells in order to calculate RMSE
I have a question about the root mean square error and Wasserstein distance on the paper https://arxiv.org/abs/2111.08736?context=stat. Consider two discrete probability distributions $P=\{P_i\}_{i=1}...
- 1
1
vote
0
answers
414
views
Area between two probability density functions as distance measure
I have two distinct probability density functions, and I would like to find a synthetic measure of how different the two distributions are. Intuitively, it would make sense to me to compute the area ...
- 11
2
votes
0
answers
155
views
Intuition behind log in kl distance
So, let's start stating that I already read both Why KL-Divergence uses "ln" in its formula? and What is the role of the logarithm in Shannon's entropy? ... However, I still have no ...
- 1,139
1
vote
1
answer
2k
views
Euclidean distance between points in PCA space along different principal component dimensions
I've picked up this project half way through, and I'm working through the last guy's code, so please bear with me.
So the original data consists of 500+ points in 150 dimensions, and I want to ...
- 31
0
votes
0
answers
140
views
Decomposing Distance Matrix D for approximating Original Matrix A
Let's say we have a matrix $A \in R^{n \times d}$ where n is the number of elements and d is the dimension size. And we calculate the pairwise distances between each elements; say cosine for instance ...
- 103
1
vote
2
answers
39
views
similarity metric for two characteristics
I am trying to develop a way to compare items. Each item has the same two properties associated with it, say, $p_{1}$ and $p_{2}$.
All I am given are these two measures for each item, each of which ...
- 31
2
votes
1
answer
664
views
How to choose a fair gamma value when performing k-prototypes clustering?
In the k-prototypes clustering algorithm, the distance function consists of two dissimilarity components - one for the numerical elements of the observations, and one for their categorical elements. ...
- 21
0
votes
1
answer
37
views
How to infer nearest neighbors using distance metrics
My team and I are trying to identify group of customers to target for an investment promotion exercise. We decided to use the control group (which already are a part of this investment exercise) and ...
- 101
2
votes
2
answers
252
views
What is the maximum number of dimensions in MDS?
If I have an arbitrary Euclidean distance matrix $D=(d_{ij}:i=1,\ldots, I; j=1,\ldots, I)$ and I want to reconstruct its elements (pairwise Euclidean distances) via classical Euclidean MDS. That is ...
- 511
1
vote
0
answers
158
views
Distance metric that is robust to collinearity
I'm trying to find a distance metric that takes into account the correlation between vectors. That is, suppose we have matrix $M$ of dimensions $n \times k$, and we take the pairwise distance between ...
- 271
2
votes
0
answers
264
views
Are the distances on a hierarchical clustering dendrogram in the same units as the input distance matrix?
I use Aitchison distance as the input to a hierarchical clustering dendrogram. I started labeling and interpreting the dendrogram but wasn't sure about a few aspects:
Are the vertical distances on ...
- 1,442
2
votes
1
answer
123
views
Which distance to assess equality between discrete distributions?
I have approximated a discrete distribution via Monte Carlo in two different ways, which metric would you use to check the distance between the two distributions? I want to use it to diagnostic ...
- 31
1
vote
2
answers
618
views
Convergence in distribution and convergence in Kolmogorov distance
Let $X, Y$ be two random variables with laws $F$ and $G$ respectively. The Kolmogorov distance between these two laws is defined as:
$$
d_{Kol}(F, G) = \sup_{x \in \mathbb R} |\mathbb P(X \leq x) - \...
- 43
1
vote
1
answer
225
views
What is the correct expression of the Hellinger Distance equation?
I am aware there are various ways to calculate the Hellinger Distance (H) depending on the context and data. One of these ways, as I understand, is via the Bhattacharyya coefficient (BC). For discrete ...
- 890
1
vote
0
answers
55
views
How to find the 'distance' between two populations?
I am somewhat new to these concepts, so please bear with me.
I have two datasets: Data set A is collected by monitoring the network data of a device when it is ...
- 33
0
votes
0
answers
287
views
using Jensen-Shannon Divergence for more than two data sets as a distance metric
I know that Jensen-Shannon Divergence for two distribution is symmetric. Is it possible to use JSD as a distance? In other words, is there a version of JSD that satisfies all the requirements for ...
- 143
4
votes
1
answer
94
views
Asymptotic value of an integral related to distances in a unit n-ball
In trying to find out the pdf of the range $T$ of euclidean distances of $m$ randomly and uniformly chosen points from the origin in an $n$-dimensional unit ball, I have obtained the following :
$$...
- 437
1
vote
1
answer
107
views
Best distance measure for cluster analysis of time series data with a circular variable
I'm doing hierarchical clustering on a set of time series (say 21 time series of 400 time points), however the variable I want to cluster is a circular variable, i.e. a directional vector between 0 ...
- 11
0
votes
0
answers
43
views
A distance function between music playlists
I am looking for a way to measure the dissimilarity/distance $d$ between a set of music playlists $\{P_i\}$ with possibly different number of songs. We may assume that a playlist contains a specific ...
- 83
1
vote
1
answer
112
views
Selecting "Statistical Distances" for "Fuzzy Match"
I am trying to better understand how to select "statistical distances" for "fuzzy matching".
To illustrate my problem, consider the following two datasets (I created this using the ...
4
votes
1
answer
491
views
Scale Invariant Statistical Distances
Problem
Suppose we have empirical distributions to two $n$-dimensional random variables $X = (X_1, X_2, ..., X_n)$ and $Y = (Y_1, Y_2, ..., Y_n)$. The goal is to find $k < n$ components, such that ...
- 143
2
votes
0
answers
53
views
Distance between hazard functions? [closed]
Let $F$ and $G$ two cumulative distribution functions with support on $(0,\infty)$. I know of many distances between distributions: Total Variation, Wasserstein, Hellinger, Kullback Liebler (...
- 21
6
votes
1
answer
700
views
What to consider when choosing between f-divergence measures? (e.g.: kl-divergence, chi-square divergence, etc.)
I have some baseline population, and I have a non random sample from that population. For both the population and the sample I have observation of some measure (for simplicity, let's say age).
I would ...
- 21.6k
1
vote
0
answers
205
views
Dissimilarity only between different treatments in dataset
I am studying the impact of wildfires on vegetation species composition. I have surveyed the burnt and unburnt areas on six sites which burned at different points in time from which I now have species ...
- 11