Questions tagged [distance]
Measure of distance between distributions or variables, such as Euclidean distance between points in n-space.
649
questions
1
vote
1answer
36 views
Intuition on Wasserstein Distance
I've been trying to familiarize myself with the Wasserstein distance and saw this answer on StackExchange by @antike that at first made a lot of sense, but then it didn't (to me, of course).
In the ...
0
votes
0answers
9 views
Using Logistic regression in record linkage
I am curious as to how logistic regression handles string variables in a training matched data set
I am aware many use Logistic regression to categorize data that includes the process of matching ...
1
vote
1answer
49 views
+50
Non-logarithmic approaches to compositional data
Background
Compositional data ($x_i>0, \sum_i x_i=c$) are usually analyzed using some kind of log-transformation (alr/clr/ilr), to take into account naturally the fact that, in presence of the sum ...
0
votes
1answer
25 views
About $D(F,G)=\int(F(x)-G(x))^2w(x)dx$ and $D(F,G)=\int(F(x)-G(x))^2w(x)dF(x)$
I learned that statistical distance between two 1-dim distributions F and G
$D_E(F,G)=\int(F(x)-G(x))^2dx$ is famous.
But what about $D(F,G)=\int(F(x)-G(x))^2w(x)dx$ or $D(F,G)=\int(F(x)-G(x))^2w(x)dF(...
0
votes
1answer
32 views
What is the name of $D(F,G)=\int(F(x)-G(x))^2dF(x)$?
Is there a statistical distance between two 1-dim distribution F and G that
$D(F,G)=\int(F(x)-G(x))^2dF(x)$?
Or to symmetrize it, take $D^s(F,G)=\int(F(x)-G(x))^2dF(x)+dG(x)$
If not, why? (What are ...
0
votes
1answer
35 views
Is it sensible to do PCA on a distance matrix?
I have 10x10 distance matrix where the distance metrics is (1 -
overlap coefficient).
I want to represent the observations in this matrix in a low dimensional space to
see how observations relate to ...
0
votes
0answers
17 views
Significance testing for Jensen–Shannon divergence?
The Jensen-Shannon divergence (JSD) measures the (dis)similarity between multiple probability distributions.
How can one determine whether the JSD of (a pair of, or multiple) distributions is ...
1
vote
1answer
26 views
Documentation or code sample for agglomerative clustering
for my undergraduate research project, I'm looking for an R code for agglomerative clustering. Basically, I need to know what happened inside hclust method in R. I have looked everywhere but don't ...
4
votes
1answer
59 views
KL-divergence for joint probability distributions?
I have a pair of joint probability distributions. I want to measure their similarity/dissimilarity. If they were single-dimensional probability distributions, then I could measure the KL-divergence or ...
0
votes
0answers
14 views
Significance test for comparing two mean Euclidean distances?
First time poster here, so please let me know if more details are needed! In short, I'm analyzing a dataset that has scores from 98 siblings from 64 families. So some families contributed only one ...
0
votes
0answers
21 views
How to simultaneously compare individuals in multiple dimensions
I have collected data about subjects in 20 different variables (all of them are continuous and have been standardized as Z scores). What I would like to do now is to obtain a valid measure of the ...
0
votes
1answer
30 views
Expected distance to nearest neighbour: Why integral of survivor function? (instead of derivative)
Previously, I asked the question how to calculate the expected distance to the nearest neighbor molecule in 3-dimensional space. This question was fully answered, which is I ask this related question ...
0
votes
0answers
13 views
Composed pair-wise similarity measure
I'm dealing with a graph theory problem for which I have calculated a series of pair-wise similarity measures (several criteria such as ancestrality, co-occurrence, sentence similarity, etc.) between ...
0
votes
0answers
18 views
Best way to evaluate ranking when one only has pairwise distances?
Let's say I have a strict ranked set of samples.
I only have a similarity measure $s$. I want to evaluate how good this similarity measure is at ranking the examples.
One approach would be to use a ...
0
votes
0answers
6 views
Patch wise feature vector comparison
I have a image of size of 64*64. I am trying to compute HOG features for the image. I have skimage for my implementation, with the following parameters:
...
0
votes
0answers
14 views
Distance measure for hierarchical nominal data
I have categorical data which follow a hierarchical structure (in fact they're medical codes). For instance:
C10: Diabetes Mellitus
E00: Senile dementia
E10: Schizophrenia
E2B1: Chronic Depression
G20:...
2
votes
1answer
22 views
Dice Distance returning nan. Workaround?
Starting Point:
I want to calculate the distances between nominal data. Furthermore, I want to see what difference it makes to only use important features (given some feature selection method). So I ...
0
votes
0answers
21 views
Quantifying if two datasets are from the same distribution, if I only have distances
Let's say that I have two datasets. The target dataset is 1K instances, the "predicted" dataset is 1M instances (but I could downsample).
I can compute the distance between instances.
How do ...
0
votes
0answers
27 views
Mantel test alternatives: linear mixed models, with row and column ids of distance matrices as random effects?
Summary: I want to model in R the relationships between pairwise spatial distances, pairwise temporal distances, and pairwise Jaccard distances, with the goal of predicting the Jaccard distance ...
1
vote
0answers
29 views
Distance to uniform distribution for continuous probability distributions [closed]
I need to quantify the distance of a continuous probability distribution $f(x)$ to the uniform distribution on $x \in (-\infty, +\infty)$. Do you know any distance functions that could help here? For ...
3
votes
2answers
160 views
What are the pros and cons of using mahalanobis distance instead of propensity scores in matching
I learned about this option of using mahalanobis distance instead of PS to do matching from the matchit() function in R. It seems a more nonparametric approach. Could you state its pros and cons and ...
0
votes
0answers
21 views
Clustering with booleans and continuous data; Gower's coefficient + PAM?
I have a medical dataset with both boolean variables and continuous variables (e.g. age/BMI). I know that clustering with K-means won't work due to the mixed datatypes. I read that I can use the Gower'...
0
votes
0answers
39 views
Calculate similarity between two matrices
I have two matrices, $A$ and $B$, each of size $n\times m$, where $n$ is discrete time points, and $m$ are the variables measured (specifically, $n$ are dates and $m$ are investments measured in ...
0
votes
0answers
14 views
Options for Clustering Analysis with Numeric & Nominal Data with Gower Distance
I am working through some cluster analysis (trying to propose new item types for various clusters). I have data that has both numeric and nominal features. After creating dummy variables for all ...
0
votes
0answers
7 views
Ranking items by the magnitude of their effect on dissimilarity?
[reposting with more detail, after previous question was removed due to lack of detail or clarity]
I am working on getting a better understanding of my company's user base. We have distinct ...
0
votes
0answers
15 views
Measuring the overlap between two probability distribution [duplicate]
I have many probability distributions, I need to compute the amount of overlap between two probability distributions. I don't know the type of distribution since it really depends on the data itself.
...
1
vote
0answers
15 views
Hausdorff Distance with Manhattan Distance
I'm applying Hausdorff Distance to understand if two datasets are representing the same subset of the space of a particular problem. The point is that I've read the Hausdorff distance computes the ...
1
vote
1answer
30 views
How do measure how different two policies are?
I have two agents that both follow a baseline behavioral policy pi(a|s). If I then modify the state-action distribution for the two agents (resulting in two new policies), is there a standard measure ...
0
votes
1answer
21 views
Is Euclidean distance the same as distance-from-correlation as $d(x, y) = \sqrt{2m[1 - r(x, y)]}$
I found in a couple of documents (e.g. this) that the Euclidean distance $d(x, y) = \sqrt{\sum_{i = 1}^{n}{(x_i - y_i)^2}}$ can be obtained from correlation coeffcient if $x$ and $y$ are standardised ...
0
votes
0answers
22 views
Similarity Metric Validation
I want to score a number of similarity metrics, i.e. given a function s(x,y) which returns a number that is higher the more similar x and y are. I'm want to objectively score a number of different ...
0
votes
0answers
31 views
Gaussian Mixture Models and distance matrix
I have a (euclidean) distance matrix and I want to perform GMM clustering.
I read in another post (gaussian mixture model - approximate a matrix) that I could apply MDS or PCA to this matrix and use ...
0
votes
0answers
43 views
Wasserstein distance for categorical data? Relationship to TVD?
Is the Wasserstein distance applicable for categorical data? e.g. if we have the distribution of different coloured balls in two bags,
...
0
votes
1answer
26 views
Is the set of distribution $\{ X | \max_t |f_X(t) - f_Y(t)| \leq \epsilon \}$ convex, where f is the cdf or inverse cdf?
I'm trying to figure out if the set is convex, where the maximum difference between cdf(or inverse cdf) of X and a reference distribution Y is smaller than $\epsilon$.
1.
Let $f_X(t)$ denote the cdf ...
0
votes
0answers
13 views
i.i.d data for computing distance correlation
When computing an estimate of the distance correlation of random vectors $X$ and $Y$ using paired samples of the vectors, should the samples be i.i.d? Can I have data in which $X_i$ and $X_j$ are ...
2
votes
1answer
30 views
Probability of points less than a fixed distance apart in a vector space
I have a distribution $D$ of points in a normed vector space (it's $\mathbb{R}^n$ using the $L_\infty$ norm, but I don't think that matters).
In this particular space, points that are less than a ...
0
votes
0answers
18 views
Computing the most representative sample of a random variable
Let $X$ be a real-valued random variable and $n > 0$. Using numerical methods, how can we find the vector $\vec v$ of $n$ real numbers that is most characteristic of $X$, in the sense that the ...
0
votes
1answer
24 views
Measure of distance between two survey responses
I've found some survey data where respondents answer 63 question by giving a response for each question between 0-10 (0 for strong disagreement, 10 for strong agreement). So I can view every ...
0
votes
1answer
73 views
Similarity/dissimilarity matrix over classes
I am trying to get something like a confusion matrix for different classes, but without training a model.
The idea is to use some kind of distance between classes.
The data set is like this, just for ...
1
vote
0answers
18 views
J-S Divergence as a Percentage
Is there a way to interpret the Jensen-Shannon divergence which is normalized to be between 0 and 1 between two probability distributions as a "percent difference", i.e., there is a x% ...
1
vote
0answers
15 views
Clustering - Distance Metric for Comparing Short Lists of Terms (non-repeating, no frequency)
Clustering involves using some distance or similarity metric.
What is the best way to score the similarity of these small sets of words? Criteria: These are technical terms which are extracted from ...
0
votes
1answer
94 views
Using $\chi^2$-Distance for Histogram-Comparison with 0-Valued Elements leads to NaNs
I want to compare two histograms by using the $\chi^2$-distance. There is a definition in the OpenCV-library. Also this question gives a lot of insight about this distance-metric. Because i didn't ...
0
votes
0answers
29 views
K-Means transform function does not match pairwise_distances_argmin_min centar calculation
I need to be able to select first N most representative points from each cluster calculated by K-means. To do so, I am aiming to calulate the distance of each point to its own cluster center and take ...
0
votes
0answers
22 views
Specific proof related to MDS distance matrix
Given a symmetric, positive semidefinite matrix A, and matrix D, where $D_{ij}=A_{ii}-2A_{ij}+A_{jj}$, prove that there exist n vectors {$\vec{v_1},...,\vec{v_n}$} such that $D_{ij}=||\vec{v_i}-\vec{...
1
vote
0answers
63 views
Product of two probability density function
Suppose $f$ and $g$ are two probability density functions. I have seen economists use $\int f(x)g(x) dx$ as some kind of similarity measure. For example, Jaffe (1986) uses sum of product of two ...
0
votes
0answers
32 views
Distance of a vector to a sample of a distribution?
Is there a better measure than Euclidean distance for my problem?
I have two functions e() and f(x). x is a vector of continuous values. The process e() outputs a variable z. Subsequently, a variable ...
1
vote
0answers
26 views
Is there a metric to identify whether samples are uniformly distributed when number of samples is small?
Suppose I have a small number of samples drawn from an unknown distribution $\{X_1,X_2,...,X_n\}$, where $0\le X_i \le L$, and $3\le n \le10$. I want to identify a metric to understand how far these ...
3
votes
0answers
79 views
Finding a sub-population from dataset matching another target dataset
Let's say one has a finite collection of i.i.d. samples from an unknown source distribution $S=\{x_{i} | i \in [1,n_{S}], x_{i} \sim p_{X_{S}}(x)\}$. Where each $x$ is multidimensional and has ...
3
votes
1answer
55 views
Distance matrix time series analysis? (Ecology/diversity)
I am trying to analyze a time series of ecological data. Each time point in the series is a matrix of animals-by-foods (that they were observed to eat). For each of these time points, I compute the ...
6
votes
1answer
34 views
two times square in distance calculation on one example?
I read a book on Kernels, See the following example. Why the authors take square two times here? what is the logic?
2
votes
0answers
40 views
Best approximation of the Mahalanobis distance by standardized Euclidean distance
I am looking for the best way to approximate the Mahalanobis distance by the standardized Euclidean distance, which would reduce the number of the required multiplications. The easiest way is the ...
