Questions tagged [distance]

Measure of distance between distributions or variables, such as Euclidean distance between points in n-space.

Filter by
Sorted by
Tagged with
1 vote
0 answers
8 views

How to calculate the max possible earth mover distance between histograms, given the buckets?

I have two histograms and am able to calculate the EMD / Wasserstein metric between them using the algorithm described here. In order to better communicate the implication of this metric, I want to ...
0 votes
0 answers
20 views

Interpreting distance based linear model

I have a dataset consisting of biological and environmental factors. My aim is to describe the physical factors influencing the invertebrate trophic composition in kelp holdfast, and compare whether ...
  • 1
4 votes
1 answer
176 views

Why do we not have a true upper limit for dissimilarity measure?

The similarity measure between two attributes always falls in the range of $[0,1]$, why is this not true in the case of dissimilarity where the value falls in the range of $[0, \infty)$? Can't the ...
0 votes
0 answers
17 views

Euclidean distance between points in high dimensions

On Wikipedia there's a statement: When a measure such as a Euclidean distance is defined using many coordinates, there is little difference in the distances between different pairs of samples. Is ...
  • 915
0 votes
0 answers
11 views

Difference-in-differences with distance sensitive treatment effect

I want to use the difference-in-difference method to estimate the effect of treatment at multiple distances to the treatment location. I have a house price dataset from 2000 to 2020 and a public ...
2 votes
1 answer
16 views

In DBSCAN, what happens if points have distance exactly equal to the Epsilon radius of a core point?

In DBSCAN the border points are points in the eps-neighborhood of a core point. But what if a point has distance exactly equal to Epsilon from a core point? Is it considered inside the eps radius, or ...
0 votes
1 answer
34 views

How do I compare multivariate normal distributions and get a p-value?

I have sample-data for two multivariate normal distributions. From this sample-data, I can calculate each distribution’s parameters (means and standard deviations). How do I quantify the distance (or ...
2 votes
0 answers
46 views

A random perturbation of a distance matrix

Let $A=(d_{ij})$ be a distance matrix, i.e., $d_{ii}=0$ for all $i$ $d_{ij}>0$ for all $i\neq j$ $d_{ij}=d_{ji}$ for all $i,j$ $d_{ij}+d_{jk}\geq d_{ik}$ for all $i,j,k$ How to find a random ...
  • 321
0 votes
0 answers
12 views

evaluation measure to assess expected value in sports betting

I'm wondering if there is a specialized evaluation measure for the expected value of bets in sports betting. For illustration, let's say we have this very specific scenario of a bet on an underdog ...
1 vote
0 answers
33 views

Expected value of inverse distance between two 3D normal distributions

Consider two independent trivariate normal random variables $X$ and $Y$. The means are non-zero and the off-diagonal elements of the covariance matrices are non-zero. $X$ and $Y$ does not follow the ...
0 votes
0 answers
8 views

Penalize an attribute of points based on their distance from a line without use of any threshold

We have a set of points and a line. Each point in the set has a weight attribute that is an integer. How we could penalize this weight based on the distance of the point from the line without using ...
0 votes
0 answers
8 views

What is a good dissimilarity on rooted ordered labeled trees with probabilities on the leaves?

Consider a rooted ordered labeled tree that is "binary" in that there are at most two children of any vertex and whose leaves are decorated with probabilities (the sum of the leaf ...
0 votes
0 answers
37 views

How to choose a heatmap distance and clustering method (with ecological data)?

I'm interested in learning about the various methods for clustering heatmaps in the context of ecology (specifically single species counts, presence/absence, % coverage and continuous environmental ...
  • 871
0 votes
2 answers
72 views

Statistical difference between curves

I have 12 curves (three replicates for each treatment), see attached picture. X=days; Y=percentage. The experiment has 2 variables: A: 4 cases; B: 3 cases. I would like to know, if there is a method ...
0 votes
0 answers
34 views

Mahalanobis distance for a vector with (also) angular data

I have a vector of data that is composed of 9 values: position in 3d orientation in 3d size in 3d. Obviously, angles are "circular" values, that is 359deg is near to 1deg, but $359 - 1 = ...
  • 1
0 votes
1 answer
54 views

Can we test if distance matrices are significantly farther apart?

I work in the field of linguistics, and my current project involves lots of distance matrices generated from language data, which measure the distance and similarity of dialects. Concretely, my ...
2 votes
2 answers
79 views

Calculating distances: Using count data or discrete probabilities?

I am looking into calculating distances between vectors for some data analysis. One question I have is whether I should use actual count data or convert to discrete probabilities. For some distances, ...
1 vote
1 answer
29 views

Distance measure between two sequences of different lengths [duplicate]

I am looking for some distance (or similarity) measure between two sequences, possibly of different lengths. Conceptually, I would like a measure with a property that $[3,1,5]$ is similar to $[3,1,4,5]...
  • 534
2 votes
1 answer
37 views

Is this Hellinger Distance expression correct?

Further to my previous post on the Hellinger Distance, there was one comment raised about there being different expressions of the Hellinger Distance. This has intrigued me. In the Encyclopedia of ...
2 votes
1 answer
64 views

Equating different forms of Hellinger Distance

For a research report, I want to show that the standard expression of the Hellinger Distance between two discrete distributions, $$H(p,q)={\sqrt{\frac{1}{2} \sum_{x \in X} \left[\sqrt{p(x)}-\sqrt{q(x)}...
1 vote
2 answers
30 views

What type of statistical test can I use to look at the influence of a factor on a frequency distribution (distance)

I'm trying to work out what test to use to see whether a factor (presence vs absence) affects a frequency distribution with distance as a predictor. Just wondering if anyone knows what the best test ...
0 votes
0 answers
15 views

How large a dataset do you need to use distance correlation?

How large a dataset do you need to use distance correlation? Distance correlation is well explained in Stackexchange post: Understanding distance correlation computations What is not clear to me is ...
2 votes
0 answers
30 views

Distance metric for vector of functions and real numbers

I want to cluster points which have the following feature vector: $$\theta = (f_1, f_2, \cdots, f_n, y_1,y_2,\cdots,y_m)$$ i.e. the first $n$ entries are functions $f_i$ ($\mathbb{R}\rightarrow \...
0 votes
0 answers
19 views

Multiple-site similarity (Sørensen) between two 2D-arrays

Suppose I have two binary 2D-arrays x and y. I wish to asses the multiple-site similarity between them. I think it should be ...
  • 101
0 votes
1 answer
99 views

Distribution of distances between points in Gaussian sample

Given a sample of $N$ points $\{\mathbf{x}_i\}$ from a bivariate Normal distribution (arbitrary mean and covariance matrix, though I'm mainly interested in the zero-mean case), how does one derive the ...
  • 101
2 votes
1 answer
107 views

For the logit model, how is the maximum likelihood and minimum distance estimator related?

Suppose I have data $\{y_i,x_i\}_{i=1}^N$, where $x_i\in\{s_1,...,s_K\}$ and follows a discrete uniform distribution. For each realized $x_i$, $y_i$ is generated by the logit model, i.e., $Pr(Y_i=1|...
  • 2,152
2 votes
2 answers
102 views

Comparing PCAs using the distances between the centroids of groups

For example, I have a data for N countries where quantitative values for men and women are given. I ran PCA separately for the data from each country. PC1 and PC2 explain most of the variance, so I ...
0 votes
0 answers
20 views

Calculate distance between generated probability distribution/point and target distribution

In my current project I have a trained neural network (pytorch-based), which is used for classifying data into n labels. Based on my current experience this is done ...
  • 101
1 vote
1 answer
20 views

Algorithm to find closest document containing a set of strings, or variations of them

I have one dataset (A) containing several fields (strings) per sample. One of these fields is a name, and the others are all alphanumeric identifiers. I have another dataset (B) which contains highly ...
  • 375
0 votes
0 answers
33 views

K-NN model with a maximum distance to be considered a neighbor

Is there some way to set a maximum distance from which a k-NN method will no longer classify an unknown sample? I am trying to build a k-NN method to classify some data. The method works good, however,...
  • 1
0 votes
0 answers
17 views

Transform Earth mover's distance/Wasserstein metric to percentance (0 to 1 or 0% to 100%) [duplicate]

Ok, I do some calculations between binned histograms via the use of Wasserstein metric and more specifically this python library.. https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats....
0 votes
0 answers
10 views

What classification/stats method can I use to find which x,y,z coordinates most likely correspond to a 2nd set of coordinates

TLDR: I need to match two sets of x,y,z coordinates of differing lengths. Although they're sampled from the same source, they do not perfectly overlap. The yellow mesh has ~7000 rows, but the blue is ~...
  • 173
0 votes
0 answers
58 views

How to reduce the number of observations (rows) by keeping the most unique (less redundant) ones

Although not a very common usecase, I need to select a subset of observations that withhold the maximum of the total variance of the dataset. In other words, I would like to remove rows that are "...
0 votes
0 answers
29 views

Hybrid correlation/MSE metric?

Apologies as this is probably a rather naive question I would like to compare two vectors and neither a correlation (e.g pearsons/spearmans) nor a distance metric (eg rmse) is quite right and ideally ...
  • 201
0 votes
0 answers
13 views

Why $P$ and $Q$ don't exist on the same coordinate, they need to be reconciled (processed) to exist on the exact same cells in order to calculate RMSE

I have a question about the root mean square error and Wasserstein distance on the paper https://arxiv.org/abs/2111.08736?context=stat. Consider two discrete probability distributions $P=\{P_i\}_{i=1}...
  • 1
1 vote
0 answers
130 views

Area between two probability density functions as distance measure

I have two distinct probability density functions, and I would like to find a synthetic measure of how different the two distributions are. Intuitively, it would make sense to me to compute the area ...
2 votes
0 answers
55 views

Intuition behind log in kl distance

So, let's start stating that I already read both Why KL-Divergence uses "ln" in its formula? and What is the role of the logarithm in Shannon's entropy? ... However, I still have no ...
1 vote
1 answer
485 views

Euclidean distance between points in PCA space along different principal component dimensions

I've picked up this project half way through, and I'm working through the last guy's code, so please bear with me. So the original data consists of 500+ points in 150 dimensions, and I want to ...
0 votes
0 answers
70 views

Decomposing Distance Matrix D for approximating Original Matrix A

Let's say we have a matrix $A \in R^{n \times d}$ where n is the number of elements and d is the dimension size. And we calculate the pairwise distances between each elements; say cosine for instance ...
0 votes
0 answers
15 views

Sorting samples from distribution before calculating the distance [duplicate]

I have to evaluate different methods for distribution fitting. So, given an sample set A I get a fitting distribution B or I might get another sample set C that is much like A. Now I need to do a ...
  • 83
0 votes
0 answers
84 views

Computing distance and standarization of features

Intro: suppose we have $n$ observations with $m$ features, represented by a $n\times m$-matrix $X$, and two specific points $x,y\in\mathbb{R}^m$, and we are interested in distances between $X$ and $x,...
1 vote
2 answers
33 views

similarity metric for two characteristics

I am trying to develop a way to compare items. Each item has the same two properties associated with it, say, $p_{1}$ and $p_{2}$. All I am given are these two measures for each item, each of which ...
0 votes
0 answers
3 views

What metric is best fitted for comparing encodings?

I am trying to compare two distributions, that each correspond to different numerical encodings, e.g. compare fp32 encodings to various other encodings on a same set of values. However I do not know ...
0 votes
0 answers
50 views

Cluster confidence scoring

I have a scenario where I am provided a list of clusters and pairwise distance only between items in same cluster. I need to rank these clusters based on some kind of relative score from this info. e....
0 votes
0 answers
23 views

Generalize Kendall tau distance to ordered groups?

If we have two ordered lists, e.g., A=[1,2,3,4,5], B=[3,5,4,2,1], then we could use Kendall tau distance (which is 0.8) to ...
0 votes
0 answers
50 views

Finding a Size Invariant Pattern in Noisy Data

I want to find similar patterns in my data, I assume that the patterns will be of different sizes both in time and in amplitude. The usual distance metrics will not work here, since the window size is ...
  • 115
1 vote
0 answers
205 views

How to choose a fair gamma value when performing k-prototypes clustering?

In the k-prototypes clustering algorithm, the distance function consists of two dissimilarity components - one for the numerical elements of the observations, and one for their categorical elements. ...
  • 11
0 votes
1 answer
32 views

How to infer nearest neighbors using distance metrics

My team and I are trying to identify group of customers to target for an investment promotion exercise. We decided to use the control group (which already are a part of this investment exercise) and ...
  • 101
1 vote
2 answers
140 views

What is the maximum number of dimensions in MDS?

If I have an arbitrary Euclidean distance matrix $D=(d_{ij}:i=1,\ldots,I; j=1,\ldots,I)$ and I want to reconstruct its elements (pairwise Euclidean distances) via classical Euclidean MDS. That is find ...
  • 457
1 vote
0 answers
115 views

Distance metric that is robust to collinearity

I'm trying to find a distance metric that takes into account the correlation between vectors. That is, suppose we have matrix $M$ of dimensions $n \times k$, and we take the pairwise distance between ...

1
2 3 4 5
14