Stack Exchange Network

Stack Exchange network consists of 174 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Measure of distance between distributions or variables, such as Euclidean distance between points in n-space.

0
votes
0answers
9 views

Robust distance measure for correlated data

I read a paper in which the authors want to compare the overall predictive accuracy of various predictors on a set of variables by using the Mahalanobis-Distance. However the data is not even ...
0
votes
0answers
16 views

what is the mathematical difference between the distances when clustering text

Suppose i have data for text clustering, there 300 000 rows text$GOODS_NAME let's do text clustering. ...
0
votes
0answers
12 views

Methods to minimize distance between features

What are some methods to minimize distances between features belonging to a cluster? I want to do this regardless of their class. Say I have a $m\times d$ feature matrix of $m$ samples, each with $d$ ...
3
votes
2answers
93 views
0
votes
1answer
19 views

Clustering by same random projection

I have $N$, $1024$-dimensional vectors. I want to cluster them by some similarity. Given the high dimensionality, standard metrics won't work. I tried a few Approximate Nearest Neighbor ...
1
vote
0answers
25 views

Compare KS test and Wasserstein distance or Earth mover's distance

Consider two sets of data points A and B. Both these data points are from mixture of unknown number of Gaussians. The mean of the Gaussians are little different for each set (there may have few ...
0
votes
0answers
13 views

Comparison of empirical discrete distributions. Pros and cons of different metrics?

I am trying to measure the dissimilarity between two empirical discrete distributions. I am aware of various distance metrics that could be used for this purpose such as Wasserstein, Bhattacharyya etc....
1
vote
0answers
15 views

Mean distance from the centre to any point in a sphere and a cylinder [closed]

What is the mean distance from the centre to any point within a sphere of radius r? What is the mean distance from the centre to any point within a cylinder of radius r and length l?
0
votes
0answers
34 views

How to quantify distance between 2 datasets?

I have a distribution $A$ (intent-to-treat population) and its subset $B \subset A$ (treated population). I learn a propensity model $P(x \in B)$ to predict treatment. Then I sample the intent-to-...
1
vote
0answers
45 views

Distance between angle distributions

I want to quantify the complexity of the street network of different cities. For each city I have the angle distribution of its streets. My hypothesis that the more complex the street network, the ...
0
votes
0answers
19 views

Determine sample size of beta distribution to detect a difference in means to a second beta distribution for a particular confidence level.

I try to understand how a former colleague of mine came up with the answer to the following question. We have two Beta distributions with the first one having $µ_1$ = 0.2, $\sigma_1 \approx 0.02$ ...
0
votes
0answers
19 views

recommendation for distance measures when clustering LIDAR data

I want to cluster LIDAR data. The goal is to find clusters that can be matched to moving vehicles. That is, ideally there should be as many distinct clusters as there are distinct vehicles contained ...
0
votes
0answers
16 views

Distance between time series and representation (Piecewise Aggregate Approximation - PAA) (R)

I am trying to come up with a way of measuring the distance between a time series and its representation in order to see how closely the representation describes the original series. I feel like there ...
0
votes
0answers
19 views

How to find Mahalanobis distance between an observation and a population with mixed data?

I have a dataset(let's call it as 'D') with multiple continuous, nominal and ordinal variables, as follows: continuous: Total sales as well as sales figures of some products by customers nominal: ...
0
votes
0answers
25 views

Comparing distances evaluated on different vector spaces

We have a dataset of I items who have been measured over two different sets of features A, with cardinality N, and B with cardinality M, and N > M. We would like to know in which feature space the ...
1
vote
0answers
34 views

Analysis ideas for difference in variances between 4 groups?

I have data for 100 different biomarkers. Each biomarker has data for 4 groups (A,B,C,D) with ~2000 observations in each group. I've read a lot about ANOVA but I'm not looking for a difference in ...
3
votes
3answers
177 views

Probability that an observation comes from population A or B?

I'm a web developer looking into some basic statistics -- pardon me if I am using the wrong jargon. :) Considering that: I have 2 populations (A and B; each have about 10,000 observations) For each ...
0
votes
1answer
28 views

Hellinger distance for two shifted log-normal distributions

If I am not mistaken, Hellinger distance between P and Q is generally given by: $$ H^2(P, Q) = \frac12 \int \left( \sqrt{dP} - \sqrt{dQ} \right)^2 .$$ If P and Q, however, are two differently ...
0
votes
0answers
9 views

How to Measure Similarity to Ground Truth? (Measuring Similarity with Features of Different Scales/Units)

I am trying to reproduce a set of ground truth data [t_start | t_end | theta_start | theta_end] (blue) A plot of my ground truth data would show a scatter of lines along the time and angular axes....
2
votes
1answer
43 views

Choosing appropriate distance metric and algorithm for clustering for any given dataset

I have been looking for an answer/guidance/pointer to this question of mine for a while. After going through many (100s actually) posts and articles, I finally found this question, where this response ...
2
votes
1answer
37 views

Distribution of distance of N-1 gamma distributed iid random variables from minimum

I have the minimum value of N iid random variables that are gamma-distributed. The parameters of the gamma distribution are known. What would be the distribution of the distance of the remaining N - 1 ...
0
votes
1answer
22 views

detect an outlier in multi-dimensions - where the number of rows is not >> number of columns

I have the following situation: a data frame with two dimensions x and y, with three "areas": an a-parametric distribution ...
1
vote
0answers
38 views

Is pairwise distance matrix useful to k-means?

The k-means implemented in scikit-learn precomputes distances but I don't how these distances are used. In its standard version, k-means is known to compute only the distances between the points and ...
5
votes
1answer
142 views

Suitable distance metric for time-series clustering with respect to location of shapes

I'm doing clustering on time-series (each time-series has the information for one day = 24 hours). For the clustering purpose, it's important for me to consider the time period in which the shape of ...
2
votes
0answers
94 views

Obtain within-group Gram matrix out of distance matrix

Gram matrix Let $\bf X$ be a n x p dataset with columns (variables) centered. Then p x p $\bf X'X$ is the total scatter matrix ...
0
votes
0answers
29 views

How to compute distances with both categorical and continuous attributes?

I have to handle with a datast containing both categorical attributes (around 25) and continuous attributes (around 25). I would like to do outliers detection. I think that it would be a good idea to ...
0
votes
1answer
14 views

Single Linkage Clustering with Manhattan metric

Say suppose we are having 5 data points with 3 attributes each ... (4,3,1) (2,1,5) (1,2,3) (2,3,1) .... Now let us build the distance matrix. If we do Manhattan metric then the cell corresponding to ...
1
vote
0answers
43 views

What happens with Mahalanobis-Distance, when the assumption of equal Covariance-Matrices breaks down

Assume that we want to compare the forecast quality of various forecasters $f$ on $n$ values such as stock-market prices or whatever. We could then define a "Mahalanobis-Distance" (MD) (or rather ...
0
votes
0answers
30 views

Common methods to calculate total distance from data with categorical, continuous and counting variables

I have a data set with categorical, continuous and counting variables. I want to be able to use a method that will give me a distance for each pairwise data point. From my understanding, each type ...
0
votes
0answers
30 views

Similarity of two sets of points

PROBLEM I have one set of 10 points, $X = \{(x_1,y_1),\,\dots,\,(x_{10},y_{10})\}$ and two sets of 3 points each, $A = \{ (a_1,b_1),\, (a_2,b_2),\, (a_3,b_3) \}$ and $C = \{(c_1,d_1),\,(c_2,d_2),\,(...
0
votes
0answers
18 views

Which is the best distance metric in an Indicator matrix

Is it okay to use the $\chi^2-distance$ when we have a indicator matrix? With Indicator matrix I mean the complete dijuntive table that is used in the Multiple Correspondence analysis. I mea n, if we ...
0
votes
1answer
64 views

Is Earth Mover Distance has maximum bound?

I have two probability distributions which each distribution has sum up to 1. I want to compute the distance between those two probability distributions. I want to use Earth Mover Distance to ...
4
votes
2answers
97 views

The Curse of high Dimension And Distance

For extracting features from video frames (2 sample/sec) I use keras framework in python and load VGG16 that input size is (150,150,3) and output size is (4,4,512). After the feature extraction step I ...
4
votes
2answers
84 views

Is the maximum bound of Euclidean distance between two probability distributions equal to $\sqrt{2}$?

I used Euclidean distance to compute the distance between two probability distribution. The example of computation shown in the Figure below. As my understanding, the maximum distance occur while $...
11
votes
3answers
1k views

What's the maximum value of Kullback-Leibler (KL) divergence

I am going to use KL divergence in my python code and I got this tutorial. On that tutorial, to implement KL divergence is quite simple. ...
6
votes
2answers
223 views

Do Autoencoders preserve distances?

Based on my understanding, autoencoders are used to find a compact representation of input features that carries the essential underlying information. Is there any relationship between the L2 ...
1
vote
0answers
67 views

Gower's Distance + PCA : How to interpret in R? [closed]

I have a dataset (40 records and 10 variables that is a combination of continuous variables and categorical variables (age, work experience, current salary, educational qualification, house in rural/...
0
votes
0answers
43 views

On the right dissimilarity matrix for discrete cluster analysis

Currently I'm having some trouble on deciding the right dissimilarity matrix for my data. After reading some previously asked questions here, I'm even left more confused. So, here is the deal. My data ...
0
votes
0answers
16 views

How well does a subsample of a large dataset represents the full set in a statistical sense?

Suppose that a large sample $ \{X_k\}_{k=1}^{n} $ (from a multivariate distribution $ X $) is given. I would like find a subsample $ \{ Y_k \}_{k=1}^{m} \subset \{X_k\}_{k=1}^{n} $ of this dataset in ...
1
vote
0answers
17 views

Distance vs Dissimilarity measure

I am reading up on distance and dissimilarity measures for my class on natural language processing and could not understand this slide. Why does the dissimilarity measure not satisfy item 3 ? What ...
0
votes
0answers
28 views

Using some kind of similarity measure to predict target variable from historical data

Let me first describe the data I have: I have aggregated CAN bus data (1 second windows, from the original 50 ms values) from 6 forklifts, and I also have measurements for these 6 trucks which ...
0
votes
0answers
44 views

Difference between the Wasserstein metric, mallows metric and Earth mover's distance

I'm really confused, is there a difference between the Wasserstein metric, mallows metric and Earth mover's distance? If yes What is it? Thank you
3
votes
2answers
101 views

Distance calculation on variables that cannot be represented in the Euclidean space

Given a variable such as number of events attended together, which is more of a multi-dimensional data how can you calculate a sort of distance between people (i.e. ...
0
votes
0answers
24 views

Gower vs Log Likelihood distance measure for mixed variables

I am working on distance measures and have used gone through the theory for two distance measure i.e. Gower distance measure and the log-likelihood distance measure. Am I correct to say that the ...
2
votes
1answer
39 views

What is a “Nominal Type Histogram” and “Shuffling Invariant Property”?

I'm reading this paper about comparing histograms ["Comprehensive Survey on Distance/Similarity Measures between Probability Density Functions" by Sung-Hyuk Cha] that writes in the introduction: ...
3
votes
2answers
121 views

Distance metric for source code

I'm trying to compare source code from multiple github projects, and in particular I'm looking for projects that include large chunks of code from other repositories, or large chunks with small ...
0
votes
0answers
24 views

How to measure distribution of values (e.g. location of dots on a line) in two groups?

Let's say I have 2 lines of equal length with dots on them. Each dot has a position (number). How can I measure if the distribution of these dots correlates (i.e. presence of hot spots or gaps without ...
0
votes
2answers
138 views

Hierarchical Clustering: What is the difference between linkages and distance measures?

Clustering algorithm defines a particular distance (correlation or euclidean) and a linkage (which, strangely some books call distance - single, complete, average or centroid). Conceptually, ...
11
votes
2answers
211 views

Does Mercer's theorem work in reverse?

A colleague has a function $s$ and for our purposes it is a black-box. The function measures the similarity $s(a,b)$ of two objects. We know for sure that $s$ has these properties: The similarity ...
1
vote
1answer
87 views

Binary distance measure

I want a metric - not an answer - for this. If I have a main binary sequence - 00100 I want a measure to tell me how far away another binary sequences 1's are relating to the sequence of interest. ...