Questions tagged [distance-functions]
Distance functions refer to functions used for quantifying the notion of distance between members of a set, or between objects.
332
questions
3
votes
1
answer
2k
views
Jeffries Matusita distance for 14 variables
I wish to perform Jeffries-Matusita distance on 14 spectral bands. Is there anyone who can help with how it is done in R?
5
votes
1
answer
1k
views
How to distance and to MDS-plot objects according their complex shape
Suppose I have four basal forms of signal (blue, purple, red, green). I also have created transition forms between each other. If you carefully look on the picture below, you can see that for example ...
- 2,252
1
vote
0
answers
56
views
Representing a distance matrix in the plane [duplicate]
I've worked with observations as vectors with both continuous and categorical variables. In both cases one can use dimensionality reduction techniques such as PCA (in the latter case through ...
- 121
2
votes
0
answers
54
views
Way to Measure Groupings Using Distances Between Individuals?
I am working on a problem that requires me to measure groupings of people. I have the location of every individual in my sample at every point in time. It's therefore trivial to calculate the ...
- 121
3
votes
0
answers
782
views
What is a good similarity measure to use when missing data is a significant issue?
I have a list of cities that I want to compare in terms of their similarity. Each city can described by a large but finite number of characteristics but most of them will have missing data for some ...
- 31
2
votes
0
answers
319
views
Calculate number of standard deviations separating two multivariate Gaussians?
Given a set of multivariate Gaussian distributions (from fitting a Gaussian mixture model) I would like to be able to calculate the likelihood that a data point drawn from one Gaussian will improperly ...
- 21
2
votes
1
answer
142
views
calculate Levenshtein Distance for web click stream data
I want to calculate Levenshtein Distance between to web click paths. I have a web page list around 500.
...
1
vote
0
answers
849
views
How does Gower distance work with free text?
The Gower distance measure is a good measure for mixed-type data (i.e., data attributes can be qualitative, categorical, ordinal or binary). But can data attributes be free-text (e.g., names of people)...
- 153
2
votes
1
answer
897
views
What is the $p$ in Cook's distance?
In the equation for Cook's distance:
$$D_i = \frac{\sum_{j=1}^{n}(\hat{y}_j - \hat{y}_{j(i)})^2}{p MSE}$$
the value of $p$ is defined as "the number of fitted parameters in the model." What does ...
- 123
345
votes
8
answers
141k
views
Why is Euclidean distance not a good metric in high dimensions?
I read that 'Euclidean distance is not a good distance in high dimensions'. I guess this statement has something to do with the curse of dimensionality, but what exactly? Besides, what is 'high ...
- 3,667
1
vote
2
answers
144
views
Is this a valid use case for Euclidean distance?
I have a set of points which is a count of links that users have clicked on :
...
- 627
1
vote
0
answers
2k
views
Negative Mahalanobis Distance
I would like to calculate a compound scores of several normal distributed continues standardized (z-score) variables.
Some of these measures are correlated, some are not. Hence, I would like to take ...
- 281
5
votes
2
answers
4k
views
Building the connection between cosine similarity and correlation in R
According to some articles (e.g. here) correlation is just a centered version of cosine similarity. I use the following code to calculate the cosine similarity matrix of the column vectors of a matrix ...
- 6,036
9
votes
1
answer
802
views
Comparing two distributions in Fourier space
There exist a number of tools that provide a distance between two continuous probability distributions. Most (semi)distances, like the Kullback-Leibler divergence, use probability density functions. ...
- 842
1
vote
1
answer
738
views
Distance between a transition matrix and an instance
I am trying to put a number to the distance of a sequence and how close it is to the original training corpus.
From the original training data, I got a markov transition matrix (TM).
So from the ...
- 425
7
votes
2
answers
5k
views
Is there an advantage to squaring dissimilarities when using Ward clustering?
Is there a reason to prefer squaring or not squaring the dissimilarities when clustering with Ward's method?
The question is motivated by the following statement in the documentation for R's ...
- 899
0
votes
1
answer
186
views
One huge cluster + small ones with vector-space model + cosine distance
I'm trying to cluster meaningfully a set of objects characterized by a vector space (bag-of-words) model. Each of those 5000 objects has 1-8 features ("words") from a set of 5500 possible. I used a ...
- 3
1
vote
0
answers
205
views
Average minimum distance between two random vectors
Let $\mathbf{y_1} =\begin{bmatrix}g_1x_1 & g_2x_1 & \dots & g_Nx_1 \end{bmatrix}$ and $\mathbf{y_2} = \begin{bmatrix} f_1x_2 & f_2x_2 & \dots & f_Nx_2\end{bmatrix}$. All the ...
- 11
1
vote
1
answer
1k
views
Silhouette scores for different distance metrics
I clustered a data set using PAM with a euclidean distance metric and a pearson correlation distance metric. The average silhouette value of the correlation clusters is higher at most points than the ...
3
votes
0
answers
892
views
What is a good technique for grouping objects based on binary or dichotomous traits?
I have a set of objects each of which has a list of traits. Data on the traits is binary: an object has a trait or does not. The number of objects that I have is moderately greater than the number ...
- 2,757
3
votes
0
answers
159
views
Facebook users similarities
I'm searching for a similarity metric such that given two Facebook users it returns a value that reflects how similar the two users are. The similarity metrics must take into account (at the same time)...
- 215
5
votes
0
answers
1k
views
Compute Shannon entropy between every row of a large, sparse matrix
I have a sparse, binary matrix of user (rows) and items (columns). Each element of this matrix is either 0 or 1:
...
- 22.7k
0
votes
0
answers
116
views
Dealing with Euclidean distance and dimension independence
I'm not very well informed in terms of distance measures.
What sort of distance measure would I use if I know that the various dimensions are not independent of each other? Because I think that ...
- 1,209
2
votes
2
answers
249
views
How to cluster users based on search terms
How can you cluster users based on what are searching for?
I'm working on an app which includes search functionality: a search box that allows a user to enter text and search the entire site. I have ...
- 627
2
votes
1
answer
2k
views
String clustering and centroid computation
I have a text file document containing a set of words strings that I want to cluster. I want to use the K-means algorithm. As a ...
- 155
3
votes
1
answer
1k
views
How to measure the distance (or divergence - not sure) between data and a probability distribution?
If I have generated a set of random data and I wish to measure how well these data fit, e.g. a uniform probability distribution, what are the standard ways to do that?
I am not very experienced with ...
- 141
2
votes
1
answer
141
views
Calculating (dis)similarity between different types of features
Disclaimer: I understand that this question is specific to the types of data, the end goal, etc. but I just wanted to get some quick tips regarding calculating dissimilarity between different types of ...
- 31
1
vote
0
answers
204
views
Limitations of using interpolated data
I have a data set that is composed of point locations in a landscape, lets call this dataset X. Some of the points in data set X need to be grouped together because they "function" together as a ...
- 11
3
votes
1
answer
14k
views
distortion function for k-means algorithm
I was reading Andrew Ng's ML lecture notes on K-mean clustering, in which the distortion function is defined as follow
$$J(c,\mu) = \sum^m_{i=1} || x^{(i)} - \mu_{c^{(i)}}||^2$$
I am puzzled about ...
- 1,813
3
votes
0
answers
215
views
Similarity Amongst Recipes Using Ingredients and Reviews/Descriptions
I'm still toying with things and just learning this, so please forgive any incorrect terminology.
My toy data set is a collection of recipes with a fairly significant overlap in ingredients. I'm ...
- 131
1
vote
0
answers
108
views
Multidimensional scaling of variables with multiple sub-features?
Let's say I have a year's worth of magazine issues (January, February, March, etc), and I want to visualize the differences among them. The classic example of multidimensional scaling (MDS) would have ...
- 11
0
votes
0
answers
62
views
How to rank users?
I have a list of movies ranked out of 5 based on their income.
I have users ranking the same movies. Not all users rank all movies.
How can I find out which users have the best correlation between ...
- 153
1
vote
0
answers
183
views
Making sense of the allele frequency weighted genetic distance
I found one formula to calculate the pairwise distance between samples according to their SNPs in the following paper: Siu, Jin and Xiong, Manifold Learning for Human Population Structure
Studies, ...
- 11
93
votes
6
answers
156k
views
Why does k-means clustering algorithm use only Euclidean distance metric?
Is there a specific purpose in terms of efficiency or functionality why the k-means algorithm does not use for example cosine (dis)similarity as a distance metric, but can only use the Euclidean norm? ...
- 1,071
0
votes
1
answer
204
views
What is the relation between Kullback loss and L1 and L2 loss? [closed]
I tried to find some relation between these distance (loss) measures, but couldn't find any references. However, I think it must something like this:
$$
\sqrt 2*D_{KL} < L_1 < L_2
$$
Is that ...
8
votes
2
answers
7k
views
The right distance for the clustering. Maybe Mahalanobis?
I have to do a cluster analysis and I'm asking which distance should I used.
I know that 99% of the clustering are made using a euclidean distance, but I heard about the Mahalanobis distance and it ...
- 393
2
votes
2
answers
243
views
Radial basis function network - G function?
I'm trying to understand Radial Basis Function Network.
I have (don' know how to write proper formatter mathematical functions here..):
$x = [ -1.0000, -0.5000, 0,0.5000,1.0000]$
$y_i = f(x_i)$
$f(...
- 467
4
votes
2
answers
5k
views
similarity measures with missing values
How to handle missing values when computing similarity (or distances)?
(I have binary feature values and do use the simple matching coefficient, but I feel that the answer to this question may be more ...
- 163
9
votes
2
answers
12k
views
Measuring distance between two empirical distributions
This is similar to the question Measuring the "distance" between two multivariate distributions, except that I want to measure distance between two data sets. I can imagine simply computing ...
- 261
0
votes
2
answers
895
views
Negative values distance model between 2 sets
I have two sets, $A$ and $B$, containing string values.
I am trying to model the relationship between sets $A$ and $B$ by computing the following distance:
$d = \frac{ \mid A \mid + \mid B \mid- \mid ...
- 155
1
vote
1
answer
3k
views
coding survey data for cosine similarity and euclidean distance?
I want to know how to code survey data such that a similarity function can be applied on it.
Say I want to use cosine similarity. All the search results and QA I've found while in my search deal only ...
- 223
1
vote
1
answer
1k
views
Measure of similarity/distance of data points in geographic space
Given two points $p_1=(x_1,y_1,t_1)$ and $p_2=(x_2,y_2,t_2)$, where $x$ and $y$ refer to the geographic coordinates in the plane, and $t$ to some measured value.
Two distance measures to evaluate the ...
- 2,632
1
vote
0
answers
357
views
Similarity distance score to remove outliers for survey data
I'm still a beginner at data mining. I'm working on finding the association rules from hypothesis X to conclusion Y. To this end, I've conducted a survey with questions that go something like this:
<...
- 223
3
votes
1
answer
923
views
Distance or Similarity metric for 2D frequency data maps
I want to compare the distance/similarity of 2D flood frequency data maps. The maps are square with YxY grid size and in each cell of the map is stored its flood frequency. For example in a 5x5 grid ...
- 33
3
votes
1
answer
684
views
Metric for nearest neighbor method
Is there a requirement that the measure used in Nearest Neighbor methods be a proper metric distance? What will happen if I use an arbitrary function (e.g., one that does not satisfy the triangle ...
- 223
4
votes
2
answers
9k
views
How to find the distance from data point to the hyperplane with MATLAB SVM?
I am using the SVMStruct function in MATLAB (with RBF kernel) to classify my data, and it works great. But now I need to compare the distance from the data points to the hyperplane, or to find the ...
- 43
5
votes
1
answer
2k
views
Covariance and correlation matrix comparison
I am aware that this question may be too broad and that answers are scattered in various posts, but i need concise and organized answer.
My dataset consists of linear measurements of cranial ...
- 597
1
vote
1
answer
987
views
Transforming confusion matrix to distance matrix
I have an annotation task which I need to analyze for accuracy, category fine tuning and reliability , there are many annotators assigning multiple set valued categories to items. And I have come up ...
- 317
3
votes
1
answer
1k
views
Log-likelihood distance measure validity for clustering
I have calculated log-likelihood distances between 50 sequences according to the Formula (1):
$$
D(X_i,X_j)= 1/2(\log p(X_i|Mod_j)+\log p(X_j|Mod_i)),
$$
where $
p(X_i|Mod_j)
$ is the likelihood ...
- 759
5
votes
2
answers
2k
views
Choice of distance metric when data is combination text/numeric/categorical
I have a large table of attributes of different real-world movie theaters. I have classified them by the "true" physical entity to which they belong, so that there may be multiple records for a given ...
- 153