Measures of closeness between distributions, clusterings, data sets or other objects.

learn more… | top users | synonyms

1
vote
2answers
20 views

how to measure similarity of two lists of continuous data with different length

I have two lists of continuous data with different length. a) How should I measure (dis)similarity of these two lists? or, as these lists can be formed into histogram, how can I quantify ...
1
vote
0answers
20 views

Cluster analysis of large, multiple answer dataset

I have to analyse data from a marketing study. I will use SPSS. The questionnaire will look like this: Q: Imagine Situation X. Select 1-3 Criteria from the list that best describe your feeling. ...
1
vote
2answers
45 views

Classifying time-series similarity - what variable should I train on?

I have ~10,000 time series, each with 65 time points. I'm interested in classifying each pair of time series as "similar" or "not similar". Here's an example of two similar (left) and not similar time ...
1
vote
0answers
17 views

Similarity between sets with different size

is there a distance measure like jaccard for sets with different sizes? For example A=['a','b','c'] and B=['a','d'] I would like to include the total intersection as well as the order. The ...
0
votes
0answers
13 views

Recommender; Matrix factorization: How to compute item to item recommendations

I am trying to compute item to item recommendations. Basically to output related/similar movies. Not depending on the user but only on the movie. My first thought was to compute the dot product for ...
0
votes
1answer
26 views

How to find similarity matrix from a confusion matrix?

How to find similarity matrix from a confusion matrix of a multiclass classification? For example in this paper: Exploiting confusion matrices for automatic generation of topic hierarchies and ...
0
votes
0answers
15 views

To what degree does Mahalanobis distance account for correlations of the data?

I understand that Mahalanobis distance is used when your data are correlated (e.g., as with many environmental variables), but just how correlated can the variables be? Should I be screening and ...
0
votes
1answer
23 views

Error when computing Cosine Similarity

I'm trying to compute the cosine similarity of the following vector: 1 1 0 1 2 0 1 1 $ \frac {1*2+1*0+0*1+1*1}{ \sqrt(1^2+1^2+0^2+1^2)\sqrt(2^2+0^2+1^2+1^2)}$ ...
0
votes
1answer
36 views

Order a correlogram

I have 30 days data, and I am plotting correlogram using corrplot (R package). I have an option to order the correlogram with hclust option and hence I obtain plot as shown on right side of below ...
1
vote
0answers
16 views

What does it mean when least squares and instrumental variables give similar results?

What can said when for some data set the result of a least squares regression is extremely close to the result of an instrumental variable regression?
0
votes
1answer
45 views

Alternate distance metrics for two time series

I have time-series data of different houses. Assume it is power consumption data. Now, I want to cluster the houses following similar power consumption pattern utmost. So, the various distance metrics ...
0
votes
0answers
18 views

Similarity measure for short series

I am looking for the best similarity measure between pairs of equal length series. These are concentrations of molecules measured in discrete fractions, the number of fractions is small (5-6). The ...
0
votes
1answer
37 views

Preprocessing survey data for clustering

I want to find 4-10 clusters in survey data with 100 questions answered by 2000 individuals using a technique such as K-means or Gaussian Mixture Models. There is no response variable so the ...
3
votes
1answer
335 views

Implicit feature space of Power Kernel

For the polynomial kernel, $K(x,y) = (x^Ty+c)^d$, the implicit feature space $\phi$ for which $K(x,y) = \phi(x)^T \phi(y)$ is of finite dimension and well known [1][2]. It is also well known that the ...
0
votes
0answers
10 views

How to find similar documents with mixed features

I am working on finding a similar documents problem. I have 3 sets of features ...
0
votes
1answer
33 views

How to find similar document with new vocabulary

I am working on a problem of finding similar documents. I am using a Tf-Idf based vector space model representation of documents and it gives me good results. However when I encounter a document ...
0
votes
0answers
14 views

Multivariate DTW

I am not able to find any resources in case of DTW for multivariate signals. For example, I would like to find DTW similarity between two objects where each object contains following signals: ...
2
votes
2answers
45 views

Proper statistical analysis for similarity of two paired datasets

I am an engineering student and as part of an undergraduate research, I created a device that measures a certain value. However, there is a commercial device that measures that same value. Here's ...
0
votes
2answers
25 views

Cluster users based on file access and application usage

I have access to file access data for employees in my organisation. For each employee I can see what read/write operations they carried out in the last month or so. Per operation, I see a user ID, ...
0
votes
0answers
18 views

Cluster logistic regression datasets

Let's say that I've got a 100 advertising campaigns and for each I'm trying to predict that a user will click on the advertisement or not using logistic regression. So basically I'm creating 100 ...
1
vote
0answers
16 views

Similarity between an item and a set

I have two items $I_0$ and $I_1$ and a set $S$. What I want to achieve is to compare which item (either $I_0$ or $I_1$) is more similar to the set. I can compare both $I_0$ and $I_1$ to every element ...
0
votes
0answers
10 views

Modelling distributions evolving with time

Say we have N probability vectors of dimensionality D describing how a process evolves in N time steps, something like multidimensional time series. What's the best way to model this sequence in ...
1
vote
1answer
42 views

Similarity measures between bimodal distributions

I have to compare two bimodal distributions. So far, I have found the Bimodality Coefficient: $$ BC = \frac{\mu_{3}^{2} + 1}{\mu_{4} + 3\frac{(n-1)^2}{(n-2)(n-3)}}, $$ with $\mu_{3}$ referring to ...
5
votes
1answer
51 views

What is the purpose of row normalization

I understand the reasoning behind column normalization, as it causes features to be weighted equally, even if they are not measured on the same scale - however, often in the nearest neighbour ...
1
vote
1answer
92 views

Similarity / clustering methods for temporal event data

I am looking for methods to calculate a reasonable similarity metric or clustering for my data. I have data in the form of a number of distinct events (lets call them A, B, C, D, ...), and the time ...
0
votes
0answers
30 views

should I count repeated items in jaccard similarity?

I have two lists: A and B where Size(A) = 4 and size(B) = 10. I would like to find the Jaccard similarity between these two lists. Suppose A = {Tom, George, John, Jennifer} and B={Tom, Jessica, ...
0
votes
1answer
49 views

similarity measure [closed]

I have some question concern similarity measure and i need your help (I'am new in statistics) Suppose that we have a matrix $M$ where $M(i, j)$ is the similarity measure between user $i$ and user $j$ ...
3
votes
2answers
41 views

Position-dependent substitution cost matrix for sequence alignment

Many sequence alignment algorithms and those which measure similarity between sequences require specification of a cost matrix defined on the sequence alphabet. However, the cost incurred does not ...
1
vote
0answers
50 views

How to measure similarity between features in different datasets, if possible?

Is it possible to measure the similarity between two different features in different datasets? It could sound like a non-sense question, but it has sense in the context of merging two almost ...
2
votes
2answers
68 views

Similarity measure between groups

Imagine there is a group of people numbered 1-100, each of which have a few numerical attributes e.g. height, weight, age. There is a small sub-group (A) which ...
0
votes
1answer
48 views

Quantification of the extent of periodicity in a time series using fractal analyses

I need metrics to quantify the extent of periodicity between of a time series (for comparison with other time series), considering the time series is almost periodic. By almost periodic I mean: if I ...
1
vote
0answers
30 views

Measure how different are two groups in their answering on 25 ordinal variables

The research context: I'm interested in usage frequency of special interest websites and its predictors. I asked individuals (n=1000) for the usage of 25 different websites in 25 items in a survey ...
0
votes
1answer
27 views

Composite similarity of GitHub projects

As a hobby, I'm building recommendation system which finds related projects on GitHub. I'm computing Jaccard index for each repository, based on users who gave stars: $$J(A, B) = \frac{ \left\vert ...
0
votes
0answers
33 views

Graph Edit Distance - strings versus matrix-->vector conversion?

I have two adjacency matrices and I want to find the graph edit distance between them. The only implementations I'm seeing (I need this fast) is edit distances in MATLAB (sigh...) for strings. I have ...
1
vote
0answers
43 views

Verify model correctness

I try to explain my question with an example. Assume we have a network of 3 computers. We can measure different entity in this system such as CPU load, disk load, Log messages and etc. (all the ...
0
votes
2answers
57 views

similarity between two sentences

I have run an experiment where users were given a distorted sentence and asked to restore it. Currently, I am interested to grade the results on the scale between 0 to 1. I'm wondering if there any ...
-1
votes
1answer
29 views

Similarity measure for observations with many variable types

I'd like to do some exploratory data analysis using a 2d plot to demonstrate possible clustering of the subjects in a study. The subjects have many variables recorded, both quantitative and ...
1
vote
0answers
193 views

Using similarity measures as distances

I have a matrix where each row corresponds to an observation with binary attributes, and I am interested in performing multidimensional scaling using cmdscale on ...
0
votes
0answers
11 views

Is there any bibliography to cite the non-metric properties of the Gower distance?

I can't find a paper confirming that Gower distance does not cover all four properties of a metric distance. Any suggestion?
2
votes
0answers
36 views

Covariance function with circular similarity property

My aim is to fit functions to covariance matrices. Furthermore I would like to have these functions positive definite. For example the figure below shows a fitted covariance matrix modeled using a ...
0
votes
0answers
32 views

Calculating “consistency” of variables in a dataset

I know am sorry in advance for the layman question :) I will try to demonstrate by an example: I have a dataset of items in a clothes store with the (dummy) variables: ...
0
votes
0answers
24 views

Binary Classification of Labels with Similar Feature Distributions

I wish to classify gene interactions as 'Validated' and 'Unvalidated' based on certain features of each interaction. Each interaction has 10 different features. However, the feature distributions of ...
1
vote
1answer
235 views

topic similarity semantic PMI between two words wikipedia

I am trying to compute pointwise mutual information (PMI) using wikipedia as data source. Given two words, PMI defines the relation between two words. The formula is as below. ...
1
vote
0answers
33 views

Correlation with stratified data structure

I have a dataset with two stratas, the first level is called block, which represents the census block. the second level is two predetermined subgroups in each blocks, each subgroup contains several ...
1
vote
0answers
59 views

Finding patterns within inconsistent data

If I have a database full of data sets that I collected from a material sample going through light sensors. Each sample is similar but the data can be inconsistent depending on many factors. I ...
2
votes
3answers
65 views

Selecting the most similar subset from an alternative dataset

Background: I have two different datasets coming from two different source. Dataset A has m features (e.g. v1,v2,v3,v4) with let us say 1 million instances. Second dataset B has n features some of ...
0
votes
1answer
66 views

How to quantify the similarity/difference between groups of selected features?

Since I have a imbalanced data to deal with, I randomly down-sample the negative data to reach balanced data for following feature selection and classification. To test feature's robustness to ...
2
votes
3answers
40 views

Cluster similarity matrix as energies

I have a symmetric score matrix and I would like to cluster the values in two dimensions rather than through a tree. Is there any method / library that would take the input matrix and treat the scores ...
0
votes
1answer
71 views

(Non-parametric) test for similarity between three or more vectors of observations?

I have three vectors each of six observations. I imagine other situations will arrive in which I might have more than three vectors with other lengths. I want a statistical test to compare them. ...
0
votes
1answer
67 views

Similarity between two small datasets

I have gone through several test's (Kolmogorov-Smirnov, t-test, u-test etc..) but I am not sure which one to use when it comes to comparing two small data sets, lets say of size 10 each. ...