Measure of distance between distributions or variables, such as Euclidean distance between points in n-space.
0
votes
0answers
9 views
Robust distance measure for correlated data
I read a paper in which the authors want to compare the overall predictive accuracy of various predictors on a set of variables by using the Mahalanobis-Distance. However the data is not even ...
0
votes
0answers
16 views
what is the mathematical difference between the distances when clustering text
Suppose i have data for text clustering, there 300 000 rows text$GOODS_NAME
let's do text clustering.
...
0
votes
0answers
12 views
Methods to minimize distance between features
What are some methods to minimize distances between features belonging to a cluster? I want to do this regardless of their class. Say I have a $m\times d$ feature matrix of $m$ samples, each with $d$ ...
3
votes
2answers
93 views
Correcting Kullback-Leibler divergence for size of datasets
We have the following implementation of KLD:
...
0
votes
1answer
19 views
Clustering by same random projection
I have $N$, $1024$-dimensional vectors. I want to cluster them by some similarity. Given the high dimensionality, standard metrics won't work. I tried a few Approximate Nearest Neighbor ...
1
vote
0answers
25 views
Compare KS test and Wasserstein distance or Earth mover's distance
Consider two sets of data points A and B. Both these data points are from mixture of unknown number of Gaussians. The mean of the Gaussians are little different for each set (there may have few ...
0
votes
0answers
13 views
Comparison of empirical discrete distributions. Pros and cons of different metrics?
I am trying to measure the dissimilarity between two empirical discrete distributions. I am aware of various distance metrics that could be used for this purpose such as Wasserstein, Bhattacharyya etc....
1
vote
0answers
15 views
Mean distance from the centre to any point in a sphere and a cylinder [closed]
What is the mean distance from the centre to any point within a sphere of radius r? What is the mean distance from the centre to any point within a cylinder of radius r and length l?
0
votes
0answers
34 views
How to quantify distance between 2 datasets?
I have a distribution $A$ (intent-to-treat population) and its subset $B \subset A$ (treated population).
I learn a propensity model $P(x \in B)$ to predict treatment.
Then I sample the intent-to-...
1
vote
0answers
45 views
Distance between angle distributions
I want to quantify the complexity of the street network of different cities.
For each city I have the angle distribution of its streets.
My hypothesis that the more complex the street network, the ...
0
votes
0answers
19 views
Determine sample size of beta distribution to detect a difference in means to a second beta distribution for a particular confidence level.
I try to understand how a former colleague of mine came up with the answer to the following question. We have two Beta distributions with the first one having $µ_1$ = 0.2, $\sigma_1 \approx 0.02$ ...
0
votes
0answers
19 views
recommendation for distance measures when clustering LIDAR data
I want to cluster LIDAR data. The goal is to find clusters that can be matched to moving vehicles. That is, ideally there should be as many distinct clusters as there are distinct vehicles contained ...
0
votes
0answers
16 views
Distance between time series and representation (Piecewise Aggregate Approximation - PAA) (R)
I am trying to come up with a way of measuring the distance between a time series and its representation in order to see how closely the representation describes the original series. I feel like there ...
0
votes
0answers
19 views
How to find Mahalanobis distance between an observation and a population with mixed data?
I have a dataset(let's call it as 'D') with multiple continuous, nominal and ordinal variables, as follows:
continuous: Total sales as well as sales figures of some products by customers
nominal: ...
0
votes
0answers
25 views
Comparing distances evaluated on different vector spaces
We have a dataset of I items who have been measured over two different sets of features A, with cardinality N, and B with cardinality M, and N > M. We would like to know in which feature space the ...
1
vote
0answers
34 views
Analysis ideas for difference in variances between 4 groups?
I have data for 100 different biomarkers. Each biomarker has data for 4 groups (A,B,C,D) with ~2000 observations in each group.
I've read a lot about ANOVA but I'm not looking for a difference in ...
3
votes
3answers
177 views
Probability that an observation comes from population A or B?
I'm a web developer looking into some basic statistics -- pardon me if I am using the wrong jargon. :)
Considering that:
I have 2 populations (A and B; each have about 10,000 observations)
For each ...
0
votes
1answer
28 views
Hellinger distance for two shifted log-normal distributions
If I am not mistaken, Hellinger distance between P and Q is generally given by:
$$
H^2(P, Q) = \frac12 \int \left( \sqrt{dP} - \sqrt{dQ} \right)^2
.$$
If P and Q, however, are two differently ...
0
votes
0answers
9 views
How to Measure Similarity to Ground Truth? (Measuring Similarity with Features of Different Scales/Units)
I am trying to reproduce a set of ground truth data
[t_start | t_end | theta_start | theta_end] (blue)
A plot of my ground truth data would show a scatter of lines along the time and angular axes....
2
votes
1answer
43 views
Choosing appropriate distance metric and algorithm for clustering for any given dataset
I have been looking for an answer/guidance/pointer to this question of mine for a while. After going through many (100s actually) posts and articles, I finally found this question, where this response ...
2
votes
1answer
37 views
Distribution of distance of N-1 gamma distributed iid random variables from minimum
I have the minimum value of N iid random variables that are gamma-distributed. The parameters of the gamma distribution are known. What would be the distribution of the distance of the remaining N - 1 ...
0
votes
1answer
22 views
detect an outlier in multi-dimensions - where the number of rows is not >> number of columns
I have the following situation: a data frame with two dimensions x and y, with three "areas": an a-parametric distribution ...
1
vote
0answers
38 views
Is pairwise distance matrix useful to k-means?
The k-means implemented in scikit-learn precomputes distances but I don't how these distances are used. In its standard version, k-means is known to compute only the distances between the points and ...
5
votes
1answer
142 views
Suitable distance metric for time-series clustering with respect to location of shapes
I'm doing clustering on time-series (each time-series has the information for one day = 24 hours). For the clustering purpose, it's important for me to consider the time period in which the shape of ...
2
votes
0answers
94 views
Obtain within-group Gram matrix out of distance matrix
Gram matrix
Let $\bf X$ be a n x p dataset with columns (variables) centered. Then p x p $\bf X'X$ is the total scatter matrix ...
0
votes
0answers
29 views
How to compute distances with both categorical and continuous attributes?
I have to handle with a datast containing both categorical attributes (around 25) and continuous attributes (around 25). I would like to do outliers detection. I think that it would be a good idea to ...
0
votes
1answer
14 views
Single Linkage Clustering with Manhattan metric
Say suppose we are having 5 data points with 3 attributes each ...
(4,3,1) (2,1,5) (1,2,3) (2,3,1) ....
Now let us build the distance matrix. If we do Manhattan metric then the cell corresponding to ...
1
vote
0answers
43 views
What happens with Mahalanobis-Distance, when the assumption of equal Covariance-Matrices breaks down
Assume that we want to compare the forecast quality of various forecasters $f$ on $n$ values such as stock-market prices or whatever. We could then define a "Mahalanobis-Distance" (MD) (or rather ...
0
votes
0answers
30 views
Common methods to calculate total distance from data with categorical, continuous and counting variables
I have a data set with categorical, continuous and counting variables. I want to be able to use a method that will give me a distance for each pairwise data point.
From my understanding, each type ...
0
votes
0answers
30 views
Similarity of two sets of points
PROBLEM
I have one set of 10 points, $X = \{(x_1,y_1),\,\dots,\,(x_{10},y_{10})\}$
and two sets of 3 points each, $A = \{ (a_1,b_1),\, (a_2,b_2),\, (a_3,b_3) \}$ and $C = \{(c_1,d_1),\,(c_2,d_2),\,(...
0
votes
0answers
18 views
Which is the best distance metric in an Indicator matrix
Is it okay to use the $\chi^2-distance$ when we have a indicator matrix? With Indicator matrix I mean the complete dijuntive table that is used in the Multiple Correspondence analysis.
I mea
n, if we ...
0
votes
1answer
64 views
Is Earth Mover Distance has maximum bound?
I have two probability distributions which each distribution has sum up to 1.
I want to compute the distance between those two probability distributions. I want to use Earth Mover Distance to ...
4
votes
2answers
97 views
The Curse of high Dimension And Distance
For extracting features from video frames (2 sample/sec) I use keras framework in python and load VGG16 that input size is (150,150,3) and output size is (4,4,512). After the feature extraction step I ...
4
votes
2answers
84 views
Is the maximum bound of Euclidean distance between two probability distributions equal to $\sqrt{2}$?
I used Euclidean distance to compute the distance between two probability distribution. The example of computation shown in the Figure below.
As my understanding, the maximum distance occur while $...
11
votes
3answers
1k views
What's the maximum value of Kullback-Leibler (KL) divergence
I am going to use KL divergence in my python code and I got this tutorial.
On that tutorial, to implement KL divergence is quite simple.
...
6
votes
2answers
223 views
Do Autoencoders preserve distances?
Based on my understanding, autoencoders are used to find a compact representation of input features that carries the essential underlying information.
Is there any relationship between the L2 ...
1
vote
0answers
67 views
Gower's Distance + PCA : How to interpret in R? [closed]
I have a dataset (40 records and 10 variables that is a combination of continuous variables and categorical variables (age, work experience, current salary, educational qualification, house in rural/...
0
votes
0answers
43 views
On the right dissimilarity matrix for discrete cluster analysis
Currently I'm having some trouble on deciding the right dissimilarity matrix for my data. After reading some previously asked questions here, I'm even left more confused. So, here is the deal. My data ...
0
votes
0answers
16 views
How well does a subsample of a large dataset represents the full set in a statistical sense?
Suppose that a large sample $ \{X_k\}_{k=1}^{n} $ (from a multivariate distribution $ X $) is given. I would like find a subsample $ \{ Y_k \}_{k=1}^{m} \subset \{X_k\}_{k=1}^{n} $ of this dataset in ...
1
vote
0answers
17 views
Distance vs Dissimilarity measure
I am reading up on distance and dissimilarity measures for my class on natural language processing and could not understand this slide. Why does the dissimilarity measure not satisfy item 3 ? What ...
0
votes
0answers
28 views
Using some kind of similarity measure to predict target variable from historical data
Let me first describe the data I have:
I have aggregated CAN bus data (1 second windows, from the original 50 ms values) from 6 forklifts, and I also have measurements for these 6 trucks which ...
0
votes
0answers
44 views
Difference between the Wasserstein metric, mallows metric and Earth mover's distance
I'm really confused, is there a difference between the Wasserstein metric, mallows metric and Earth mover's distance?
If yes What is it?
Thank you
3
votes
2answers
101 views
Distance calculation on variables that cannot be represented in the Euclidean space
Given a variable such as number of events attended together, which is more of a multi-dimensional data how can you calculate a sort of distance between people (i.e. ...
0
votes
0answers
24 views
Gower vs Log Likelihood distance measure for mixed variables
I am working on distance measures and have used gone through the theory for two distance measure i.e. Gower distance measure and the log-likelihood distance measure.
Am I correct to say that the ...
2
votes
1answer
39 views
What is a “Nominal Type Histogram” and “Shuffling Invariant Property”?
I'm reading this paper about comparing histograms ["Comprehensive Survey on Distance/Similarity Measures between Probability Density
Functions" by Sung-Hyuk Cha] that writes in the introduction:
...
3
votes
2answers
121 views
Distance metric for source code
I'm trying to compare source code from multiple github projects, and in particular I'm looking for projects that include large chunks of code from other repositories, or large chunks with small ...
0
votes
0answers
24 views
How to measure distribution of values (e.g. location of dots on a line) in two groups?
Let's say I have 2 lines of equal length with dots on them. Each dot has a position (number). How can I measure if the distribution of these dots correlates (i.e. presence of hot spots or gaps without ...
0
votes
2answers
138 views
Hierarchical Clustering: What is the difference between linkages and distance measures?
Clustering algorithm defines a particular distance (correlation or euclidean) and a linkage (which, strangely some books call distance - single, complete, average or centroid). Conceptually, ...
11
votes
2answers
211 views
Does Mercer's theorem work in reverse?
A colleague has a function $s$ and for our purposes it is a black-box. The function measures the similarity $s(a,b)$ of two objects.
We know for sure that $s$ has these properties:
The similarity ...
1
vote
1answer
87 views
Binary distance measure
I want a metric - not an answer - for this.
If I have a main binary sequence - 00100
I want a measure to tell me how far away another binary sequences 1's are relating to the sequence of interest.
...
