Questions tagged [distance]
Measure of distance between distributions or variables, such as Euclidean distance between points in n-space.
700
questions
1
vote
0
answers
8
views
How to calculate the max possible earth mover distance between histograms, given the buckets?
I have two histograms and am able to calculate the EMD / Wasserstein metric between them using the algorithm described here. In order to better communicate the implication of this metric, I want to ...
- 31
0
votes
0
answers
20
views
Interpreting distance based linear model
I have a dataset consisting of biological and environmental factors. My aim is to describe the physical factors influencing the invertebrate trophic composition in kelp holdfast, and compare whether ...
- 1
4
votes
1
answer
176
views
Why do we not have a true upper limit for dissimilarity measure?
The similarity measure between two attributes always falls in the range of $[0,1]$, why is this not true in the case of dissimilarity where the value falls in the range of $[0, \infty)$?
Can't the ...
0
votes
0
answers
17
views
Euclidean distance between points in high dimensions
On Wikipedia there's a statement:
When a measure such as a Euclidean distance is defined using many coordinates, there is little difference in the distances between different pairs of samples.
Is ...
- 915
0
votes
0
answers
11
views
Difference-in-differences with distance sensitive treatment effect
I want to use the difference-in-difference method to estimate the effect of treatment at multiple distances to the treatment location. I have a house price dataset from 2000 to 2020 and a public ...
- 1
2
votes
1
answer
16
views
In DBSCAN, what happens if points have distance exactly equal to the Epsilon radius of a core point?
In DBSCAN the border points are points in the eps-neighborhood of a core point. But what if a point has distance exactly equal to Epsilon from a core point? Is it considered inside the eps radius, or ...
- 23
0
votes
1
answer
34
views
How do I compare multivariate normal distributions and get a p-value?
I have sample-data for two multivariate normal distributions. From this sample-data, I can calculate each distribution’s parameters (means and standard deviations).
How do I quantify the distance (or ...
2
votes
0
answers
46
views
A random perturbation of a distance matrix
Let $A=(d_{ij})$ be a distance matrix, i.e.,
$d_{ii}=0$ for all $i$
$d_{ij}>0$ for all $i\neq j$
$d_{ij}=d_{ji}$ for all $i,j$
$d_{ij}+d_{jk}\geq d_{ik}$ for all $i,j,k$
How to find a random ...
- 321
0
votes
0
answers
12
views
evaluation measure to assess expected value in sports betting
I'm wondering if there is a specialized evaluation measure for the expected value of bets in sports betting. For illustration, let's say we have this very specific scenario of a bet on an underdog ...
1
vote
0
answers
33
views
Expected value of inverse distance between two 3D normal distributions
Consider two independent trivariate normal random variables $X$ and $Y$. The means are non-zero and the off-diagonal elements of the covariance matrices are non-zero. $X$ and $Y$ does not follow the ...
- 216
0
votes
0
answers
8
views
Penalize an attribute of points based on their distance from a line without use of any threshold
We have a set of points and a line. Each point in the set has a weight attribute that is an integer. How we could penalize this weight based on the distance of the point from the line without using ...
0
votes
0
answers
8
views
What is a good dissimilarity on rooted ordered labeled trees with probabilities on the leaves?
Consider a rooted ordered labeled tree that is "binary" in that there are at most two children of any vertex and whose leaves are decorated with probabilities (the sum of the leaf ...
- 141
0
votes
0
answers
37
views
How to choose a heatmap distance and clustering method (with ecological data)?
I'm interested in learning about the various methods for clustering heatmaps in the context of ecology (specifically single species counts, presence/absence, % coverage and continuous environmental ...
- 871
0
votes
2
answers
72
views
Statistical difference between curves
I have 12 curves (three replicates for each treatment), see attached picture. X=days; Y=percentage.
The experiment has 2 variables: A: 4 cases; B: 3 cases.
I would like to know, if there is a method ...
- 21
0
votes
0
answers
34
views
Mahalanobis distance for a vector with (also) angular data
I have a vector of data that is composed of 9 values:
position in 3d
orientation in 3d
size in 3d.
Obviously, angles are "circular" values, that is 359deg is near to 1deg, but $359 - 1 = ...
- 1
0
votes
1
answer
54
views
Can we test if distance matrices are significantly farther apart?
I work in the field of linguistics, and my current project involves lots of distance matrices generated from language data, which measure the distance and similarity of dialects. Concretely, my ...
2
votes
2
answers
79
views
Calculating distances: Using count data or discrete probabilities?
I am looking into calculating distances between vectors for some data analysis. One question I have is whether I should use actual count data or convert to discrete probabilities.
For some distances, ...
- 67
1
vote
1
answer
29
views
Distance measure between two sequences of different lengths [duplicate]
I am looking for some distance (or similarity) measure between two sequences, possibly of different lengths. Conceptually, I would like a measure with a property that $[3,1,5]$ is similar to $[3,1,4,5]...
- 534
2
votes
1
answer
37
views
Is this Hellinger Distance expression correct?
Further to my previous post on the Hellinger Distance, there was one comment raised about there being different expressions of the Hellinger Distance. This has intrigued me.
In the Encyclopedia of ...
- 67
2
votes
1
answer
64
views
Equating different forms of Hellinger Distance
For a research report, I want to show that the standard expression of the Hellinger Distance between two discrete distributions,
$$H(p,q)={\sqrt{\frac{1}{2} \sum_{x \in X} \left[\sqrt{p(x)}-\sqrt{q(x)}...
- 67
1
vote
2
answers
30
views
What type of statistical test can I use to look at the influence of a factor on a frequency distribution (distance)
I'm trying to work out what test to use to see whether a factor (presence vs absence) affects a frequency distribution with distance as a predictor.
Just wondering if anyone knows what the best test ...
- 11
0
votes
0
answers
15
views
How large a dataset do you need to use distance correlation?
How large a dataset do you need to use distance correlation?
Distance correlation is well explained in Stackexchange post:
Understanding distance correlation computations
What is not clear to me is ...
- 101
2
votes
0
answers
30
views
Distance metric for vector of functions and real numbers
I want to cluster points which have the following feature vector:
$$\theta = (f_1, f_2, \cdots, f_n, y_1,y_2,\cdots,y_m)$$
i.e. the first $n$ entries are functions $f_i$ ($\mathbb{R}\rightarrow \...
- 225
0
votes
0
answers
19
views
Multiple-site similarity (Sørensen) between two 2D-arrays
Suppose I have two binary 2D-arrays x and y. I wish to asses the multiple-site similarity between them. I think it should be ...
- 101
0
votes
1
answer
99
views
Distribution of distances between points in Gaussian sample
Given a sample of $N$ points $\{\mathbf{x}_i\}$ from a bivariate Normal distribution (arbitrary mean and covariance matrix, though I'm mainly interested in the zero-mean case), how does one derive the ...
- 101
2
votes
1
answer
107
views
For the logit model, how is the maximum likelihood and minimum distance estimator related?
Suppose I have data $\{y_i,x_i\}_{i=1}^N$, where $x_i\in\{s_1,...,s_K\}$ and follows a discrete uniform distribution. For each realized $x_i$, $y_i$ is generated by the logit model, i.e., $Pr(Y_i=1|...
- 2,152
2
votes
2
answers
102
views
Comparing PCAs using the distances between the centroids of groups
For example, I have a data for N countries where quantitative values for men and women are given. I ran PCA separately for the data from each country. PC1 and PC2 explain most of the variance, so I ...
- 21
0
votes
0
answers
20
views
Calculate distance between generated probability distribution/point and target distribution
In my current project I have a trained neural network (pytorch-based), which is used for classifying data into n labels. Based on my current experience this is done ...
- 101
1
vote
1
answer
20
views
Algorithm to find closest document containing a set of strings, or variations of them
I have one dataset (A) containing several fields (strings) per sample. One of these fields is a name, and the others are all alphanumeric identifiers.
I have another dataset (B) which contains highly ...
- 375
0
votes
0
answers
33
views
K-NN model with a maximum distance to be considered a neighbor
Is there some way to set a maximum distance from which a k-NN method will no longer classify an unknown sample?
I am trying to build a k-NN method to classify some data. The method works good, however,...
- 1
0
votes
0
answers
17
views
Transform Earth mover's distance/Wasserstein metric to percentance (0 to 1 or 0% to 100%) [duplicate]
Ok, I do some calculations between binned histograms via the use of Wasserstein metric and more specifically this python library.. https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats....
- 137
0
votes
0
answers
10
views
What classification/stats method can I use to find which x,y,z coordinates most likely correspond to a 2nd set of coordinates
TLDR: I need to match two sets of x,y,z coordinates of differing lengths. Although they're sampled from the same source, they do not perfectly overlap. The yellow mesh has ~7000 rows, but the blue is ~...
- 173
0
votes
0
answers
58
views
How to reduce the number of observations (rows) by keeping the most unique (less redundant) ones
Although not a very common usecase, I need to select a subset of observations that withhold the maximum of the total variance of the dataset.
In other words, I would like to remove rows that are "...
0
votes
0
answers
29
views
Hybrid correlation/MSE metric?
Apologies as this is probably a rather naive question
I would like to compare two vectors and neither a correlation (e.g pearsons/spearmans) nor a distance metric (eg rmse) is quite right and ideally ...
- 201
0
votes
0
answers
13
views
Why $P$ and $Q$ don't exist on the same coordinate, they need to be reconciled (processed) to exist on the exact same cells in order to calculate RMSE
I have a question about the root mean square error and Wasserstein distance on the paper https://arxiv.org/abs/2111.08736?context=stat. Consider two discrete probability distributions $P=\{P_i\}_{i=1}...
- 1
1
vote
0
answers
130
views
Area between two probability density functions as distance measure
I have two distinct probability density functions, and I would like to find a synthetic measure of how different the two distributions are. Intuitively, it would make sense to me to compute the area ...
- 11
2
votes
0
answers
55
views
Intuition behind log in kl distance
So, let's start stating that I already read both Why KL-Divergence uses "ln" in its formula? and What is the role of the logarithm in Shannon's entropy? ... However, I still have no ...
1
vote
1
answer
485
views
Euclidean distance between points in PCA space along different principal component dimensions
I've picked up this project half way through, and I'm working through the last guy's code, so please bear with me.
So the original data consists of 500+ points in 150 dimensions, and I want to ...
- 31
0
votes
0
answers
70
views
Decomposing Distance Matrix D for approximating Original Matrix A
Let's say we have a matrix $A \in R^{n \times d}$ where n is the number of elements and d is the dimension size. And we calculate the pairwise distances between each elements; say cosine for instance ...
0
votes
0
answers
15
views
Sorting samples from distribution before calculating the distance [duplicate]
I have to evaluate different methods for distribution fitting. So, given an sample set A I get a fitting distribution B or I might get another sample set C that is much like A. Now I need to do a ...
- 83
0
votes
0
answers
84
views
Computing distance and standarization of features
Intro: suppose we have $n$ observations with $m$ features, represented by a $n\times m$-matrix $X$, and two specific points $x,y\in\mathbb{R}^m$, and we are interested in distances between $X$ and $x,...
1
vote
2
answers
33
views
similarity metric for two characteristics
I am trying to develop a way to compare items. Each item has the same two properties associated with it, say, $p_{1}$ and $p_{2}$.
All I am given are these two measures for each item, each of which ...
- 31
0
votes
0
answers
3
views
What metric is best fitted for comparing encodings?
I am trying to compare two distributions, that each correspond to different numerical encodings, e.g. compare fp32 encodings to various other encodings on a same set of values.
However I do not know ...
0
votes
0
answers
50
views
Cluster confidence scoring
I have a scenario where I am provided a list of clusters and pairwise distance only between items in same cluster. I need to rank these clusters based on some kind of relative score from this info.
e....
0
votes
0
answers
23
views
Generalize Kendall tau distance to ordered groups?
If we have two ordered lists, e.g., A=[1,2,3,4,5], B=[3,5,4,2,1], then we could use Kendall tau distance (which is 0.8) to ...
- 111
0
votes
0
answers
50
views
Finding a Size Invariant Pattern in Noisy Data
I want to find similar patterns in my data, I assume that the patterns will be of different sizes both in time and in amplitude. The usual distance metrics will not work here, since the window size is ...
- 115
1
vote
0
answers
205
views
How to choose a fair gamma value when performing k-prototypes clustering?
In the k-prototypes clustering algorithm, the distance function consists of two dissimilarity components - one for the numerical elements of the observations, and one for their categorical elements. ...
- 11
0
votes
1
answer
32
views
How to infer nearest neighbors using distance metrics
My team and I are trying to identify group of customers to target for an investment promotion exercise. We decided to use the control group (which already are a part of this investment exercise) and ...
- 101
1
vote
2
answers
140
views
What is the maximum number of dimensions in MDS?
If I have an arbitrary Euclidean distance matrix $D=(d_{ij}:i=1,\ldots,I; j=1,\ldots,I)$ and I want to reconstruct its elements (pairwise Euclidean distances) via classical Euclidean MDS. That is find ...
- 457
1
vote
0
answers
115
views
Distance metric that is robust to collinearity
I'm trying to find a distance metric that takes into account the correlation between vectors. That is, suppose we have matrix $M$ of dimensions $n \times k$, and we take the pairwise distance between ...
- 231