Measures of closeness between distributions, clusterings, data sets or other objects.

learn more… | top users | synonyms

0
votes
0answers
10 views

Binary Classification of Labels with Similar Feature Distributions

I wish to classify gene interactions as 'Validated' and 'Unvalidated' based on certain features of each interaction. Each interaction has 10 different features. However, the feature distributions of ...
0
votes
0answers
8 views

topic similarity semantic PMI between two words wikipedia

I am trying to compute pointwise mutual information (PMI) using wikipedia as data source. Given two words, PMI defines the relation between two words. The formula is as below. ...
1
vote
0answers
12 views

Correlation with stratified data structure

I have a dataset with two stratas, the first level is called block, which represents the census block. the second level is two predetermined subgroups in each blocks, each subgroup contains several ...
1
vote
0answers
29 views

Finding patterns within inconsistent data

If I have a database full of data sets that I collected from a material sample going through light sensors. Each sample is similar but the data can be inconsistent depending on many factors. I ...
2
votes
3answers
49 views

Selecting the most similar subset from an alternative dataset

Background: I have two different datasets coming from two different source. Dataset A has m features (e.g. v1,v2,v3,v4) with let us say 1 million instances. Second dataset B has n features some of ...
0
votes
1answer
30 views

How to quantify the similarity/difference between groups of selected features?

Since I have a imbalanced data to deal with, I randomly down-sample the negative data to reach balanced data for following feature selection and classification. To test feature's robustness to ...
2
votes
3answers
29 views

Cluster similarity matrix as energies

I have a symmetric score matrix and I would like to cluster the values in two dimensions rather than through a tree. Is there any method / library that would take the input matrix and treat the scores ...
0
votes
1answer
25 views

(Non-parametric) test for similarity between three or more vectors of observations?

I have three vectors each of six observations. I imagine other situations will arrive in which I might have more than three vectors with other lengths. I want a statistical test to compare them. ...
0
votes
1answer
37 views

Similarity between two small datasets

I have gone through several test's (Kolmogorov-Smirnov, t-test, u-test etc..) but I am not sure which one to use when it comes to comparing two small data sets, lets say of size 10 each. ...
1
vote
0answers
10 views

Testing if two rankings are equivalent

I have some process A that assigns values y_i for i=1:N to instances x_i. ...
1
vote
0answers
32 views

What are some applications of the Apriori algorithm?

My understanding is that for online promotions the Apriori algorithm is not used as much as say collaborative filtering algorithms largely because the cost of marketing to the long tail is so cheap ...
0
votes
0answers
10 views

Modify dot product for dimension independence

The distribution of the pair-wise dot products of uniformly distributed points on the unit sphere depends upon the dimension n (see, e.g. here). What alternative similarity measures exist that do not ...
1
vote
0answers
10 views

Fisher kernel or KL divergence?

I am currently reading about pLSA and LDA and how can I apply these methods on calculating document similarity. I got a feeling that common similarity measure used in pLSA is Fisher kernel, but for ...
0
votes
0answers
17 views

Diversity Index for binomial data

I have community ecology data where each sample unit has species lists for each of N subsamples. So rather than a raw count of the number of individuals of each species type at each sample unit, I ...
2
votes
1answer
35 views

A similarity measure with binary data: does this one have a name?

There are many binary similarity measures (e.g. Jaccard, Sorensen, etc), each of them is sensitive to different properties of the compared sets. I would like to use the metric $S=\frac{N_{A\bigcap ...
0
votes
0answers
8 views

What is the best practice to deal with NA values when calculating a dissimilarity matrix?

I need to calculate a matrix of distances between sites where different variables were measured. I will use it in a cluster analysis. The following is a sample of the matrix I am dealing with: ...
1
vote
1answer
48 views

Proving similarity between two rankings

We are seeking a comparison of two different rankings of the same 3 systems Background: We have conducted a test of 3 different systems. Participants experienced a system and rated it on a 7-point ...
0
votes
2answers
74 views

What is the best algorithm to find similar text documents?

I have many text documents and I would like to find similar documents to each document within my data set. Is Latent Dirichlet Allocation (LDA) the best way to do that, or are there other algorithms ...
1
vote
1answer
58 views

Comparing undirected weighted graphs

I want to get a sense of the similarity between weighted, undirected graphs. My data are from an EEG experiment, where every vertex is an electrode, and every edge is the connectivity between two ...
0
votes
1answer
49 views

Which (dis)similarity index to choose for cluster analysis?

I have data that refer to the number of occurrences of specific variable in samples: ...
1
vote
0answers
19 views

Similarity Measure for small strings

I am looking for a good similarity measure to conflate entries of a column (Product Brand in my scenario) Text like : ("Dell", "Dell Laptops"), ("ACP","ACP by XYZ"),("Acer Notebooks","Acer ...
0
votes
1answer
15 views

Is there a distance algorithm similar to Jaccard distance that handles scalar data?

we have the characteristics and (scalar) values of those characteristics for three (or more) people: ...
0
votes
0answers
11 views

Similarity of linear subgraphs

I have a directed graph, on which I have a number of linear subgraphs. In the simple example graph below, these subgraph might be a->b->c->e->g->h, ...
1
vote
0answers
12 views

Feature Extraction for landscape images

I work on a landscape-image database (forests, beaches, landmarks, cities etc.) and I'm trying to find the similarity between them, I don't have labels or any specific classes, I just want to find ...
2
votes
1answer
85 views

Overlap between two normal pdfs [duplicate]

I have two normally distributed random variables (estimated from two different sets of samples), and I'd like to know how "similar" those variables are (in order to compare the sets). I had the idea ...
5
votes
2answers
68 views

How to find weights for a dissimiliarity measure

I want to learn (deduce) attribute weights for my dissimilarity measure that I can use for clustering. I have some examples $(a_i,b_i)$ of pairs of objects that are "similar" (should be in the same ...
0
votes
0answers
21 views

Detect rhymes in lyrics

I am testing several features computed from a dataset with several machine learning algorithms, and I would like to see if rhymes contribute to the genre classification of lyrics. Therefore, I would ...
2
votes
1answer
44 views

Turn a distance measure into a kernel function

I have read here that an easy way to turn a distance function $d$ into a similarity function $s$ is to compute: $s = e^{-\gamma * d}$. I believe that this is also what is done with the RBF kernel. ...
1
vote
0answers
12 views

Similarity/duplicate measure for collection of vectors

I have 12 vectors of the size 1x16, which are generated as a side-product of my algorithm. If any of the vectors are very similar, that could indicate that my algorithm is performing badly. Roughly ...
2
votes
1answer
34 views

How to measure similarity of two rectangles' ratios? [closed]

I have two rectangles $R_1=(w_1,h_1)$ and $R_2=(w_2,h_2)$ and I would like to check how similar their ratios are. My first naive thought was this: ...
0
votes
0answers
35 views

Similarity among multivariate time series vs similarity among univariate time series

I am trying to generalize the similarity among univariate time series into multivariate time series. Let's say there are 2 multivariate time series data X and Y: X1, X2, X3 in first multivariate ...
0
votes
0answers
21 views

Using PERMANOVA or BIOENV or both?

I just came across these two methods for analyzing community data and their response to environmental gradients: PERMANOVA (adonis function, package vegan) and bioenv (vegan). As far as I understood, ...
0
votes
0answers
13 views

Determining similarity among 4 datasets based on demographic composition

I have 4 data sets and I am trying to determine what is the order of similarity between Set XY and Sets A, B, or C based on audience characteristics (in other words, which site has the the most ...
6
votes
3answers
173 views

Quantifying similarity between two data sets

Summary: Trying to find the best method summarize the similarity between two aligned data sets of data using a single value. Details: My question is best explained with a diagram. The graphs below ...
4
votes
1answer
95 views

Get k most diverse objects from dendrogram (hierarchical clustering)

I have a dendrogram which groups similar object in a hierarchical order. The problem I try to solve is based on a dendrogram how to get k most diverse objects. E.g. We start with some random (?) ...
0
votes
0answers
17 views

similarities between two signal using correlation

let say we have two signal ,there is given their time domain plots and also correlation plot and final correlation plot it should be noted that both signal have same spectral structure,they ...
0
votes
2answers
287 views

TF-IDF versus Cosine Similarity in Document Search

I'm wondering if anyone can help me out or point out some resources to learn more about TF-IDF and document search. I'm trying to implement a basic document search and am trying to better understand ...
0
votes
0answers
25 views

A good metric for measuring the “similarity” between a number and a series of numbers of unknown distribution

I'm very new to stats so please excuse me if some of my terminologies are wrong. I have a series of N numbers, and I don't know the distribution of these numbers. Now, I have another number K, and I ...
0
votes
0answers
28 views

Correlation/similarity binary time series

Which are the best or most used methods to find similarities between binary time series? ...
3
votes
2answers
99 views

How to measure similarity of bivariate probability distributions?

I have three different distributions of 2D data: or Now I like to know whether distribution two is more similar to distribution one (2 to 1) than distribution three is to distribution one (3 to ...
1
vote
0answers
42 views

recommender - recommendation reasoning (get similarities )

I’m using recommenderlab to build a UBCF and produce recommendations. The process seems to be ok, and the predictions are making sense. Now,what I need to have, is the reasoning for each prediction, ...
1
vote
0answers
65 views

Similarity metric for 2 sets of vectors

I'm trying to determine the similarity between two sentences. I have vectors for each word in a corpus, and using cosine distance of the two vectors, I can get quite a good "similarity" score ...
2
votes
1answer
42 views

Formulation of null and alternative to show two sets have similar mean

I have two sets of ratings between 1 to 5. The second set is generated using an algorithm, and I am currently trying to find the best set of parameters, so the ratings are similar to my first set. ...
3
votes
0answers
20 views

Measuring “almost the same” when significantly different

The following table is of UK domiciled university applicants to UK universities in $2013$ and of acceptances, grouped by self-declared disability. A $\chi^2$ test on the numbers shows an extremely ...
0
votes
0answers
43 views

how to measure the similarity or overlapping significance between two character vectors with pre-ranked items?

Here there are two character vectors (like V1 and V2). Items in each vector are unique, which means no duplicate in each vector. All the items in the vectors come from a known set (#item in set >> ...
0
votes
1answer
41 views

MinHashing and Hash Digests/Signatures

Nilsimsa and SimHash are two similarity hash algorithms that produce a single hash digest/signature per document. Comparing two documents is as simple as computing the hamming distance between two ...
2
votes
1answer
46 views

Compare two unnormalized density functions given the values at samples

I have the density values of two unnormalized density functions $p$ and $q$ at 2000 points: $\mathbf{p} = (p(x_1), p(x_2), \dots p(x_{2000}))$, $\mathbf{q} = (q(x_1), q(x_2), \dots q(x_{2000}))$. Now ...
0
votes
1answer
68 views

Similarity scoring to compare multi-dimensional datasets

I am trying to come up with a mechanism of scoring a set of multidimensional datasets based on a similarity with an ideal dataset. Each dataset will all have the same dimensions along with the ideal. ...
0
votes
0answers
151 views

How to measure consistency or simiarity between two data sets?

I am working on a study that elicits priorities (weights) from experts on a set of alternatives using different methods. The snapshot below shows the resulting weights, which sum up to 1 for each ...
1
vote
0answers
15 views

Similarity measure of lower and upper triangle matrix [closed]

Do you have any idea to create index regarding the similarity of lower and upper elements (i-jth element and j-i th element) of triangle matrix A? One way is given the index as Tij = A[ij] / A[ji] ...