Measures of closeness between distributions, clusterings, data sets or other objects.

learn more… | top users | synonyms

1
vote
1answer
14 views

How to evaluate level of significance of two similar correlations?

I have four datasets: A1, A2, B1, B2. Every dataset has between 100-300 items. Every item in every dataset has two values: x, y The goal: Find what datasets have similar x values. If the datasets ...
-1
votes
0answers
12 views

Similarity between two large vector

I have a two large/long vectors which I try to calculate the similarity between them. First vector: Length of the vector nearly: ~990 < ["RAB31", 25], ["SNAP25", 34], ["PCK2", 34], ["HMGCR", ...
0
votes
1answer
18 views

Assessing correlation/similarity in shape and position of two datasets

I have data-sets within which position and shape (curves, relative increase/decrease at any given x-value) are the defining characteristics, such as: And a simulation which attempts to recapture ...
1
vote
2answers
40 views

What is maximum and its computation in the function dist() {stats} in R?

In R, we can calculate a distance matrix using the method "maximum" in the function dist() in the stats package. ...
0
votes
0answers
14 views

Using frequency tables to judge similarity

I'm a newbie to data mining, and attempting to simplify a problem. Assume I want to find out what skills make up a certain profession. So I'll ask the user to input their profession, followed by a ...
3
votes
1answer
58 views

Finding most similar observations in two data sets

I have the following question. Given are two data sets A and B with around 10,000 observations described by different variables categorical and numeric. The task is to find these observations in B ...
0
votes
1answer
51 views

distance measure of two discrete probability histograms (distance between two vectors)

I have multiple sets of discrete probability histograms(vectors) and I want to measure the distance between each histogram. I have done some research but I am in doubt. Literature suggest I could ...
0
votes
0answers
19 views

Numerical Similarity Measures for Dirty Data

I need to compare two datasets to see if they are "similar" in terms of numerical similarity between given columns. The problem is, however, that I do not know the characteristics of the 2nd dataset. ...
0
votes
1answer
21 views

How to gauge and specify similarity of energy data for two US states?

I am working on a research project to gauge the potential for rooftop photo-voltaic solar power in Wisconsin in the years 2020 and 2030. Part of this research involves analyzing hourly electric usage ...
-1
votes
1answer
44 views

proper Similarity measure and clustering algorithm for binary data

data sample as follow : interest to find clusters of similar users, pages number around 100 pages. users around 1000 , i would like to know what are proper Similarity/dis. measure can used in this ...
1
vote
0answers
83 views

Outlier similarity

I have a group of sensors where each sensor records 10 different binary data points which is then saved as an integer that can range from 0 to 2^10 - 1. To make sure that's clear, to put it another ...
0
votes
0answers
25 views

Clustering subjects regarding binary property vectors

My data set consists of 120 subjects and 50 binary attributes. I want to find clusters in that. So far I started by a visual analysis plotting the n x n matrix taking the several similarity measures ...
0
votes
1answer
54 views

How to soften or mitigate vector similarity measure?

I would like to evaluate a similarity between two objects X and Y by comparing a neighbourhood in which they're located. I construct two sets of nine concentric and equidistant circles with centers ...
1
vote
1answer
27 views

Selecting the number of hashes for minhash? Working with extremely sparse data and want more collisions

I'm attempting to use minhash to generate clusters and similarities, and I am primarily using ideas from these resources. http://www2007.org/papers/paper570.pdf ...
0
votes
0answers
19 views

Recommender system - combining collaborative filtering and content based

I read a lot of stuffs about combining collaborative filtering and content-based but no one seems to bother explaining how to get those so-called hybrid recommenders or they do it so shallowly that I ...
0
votes
0answers
14 views

Is spatial autocorrelation and similarity the same?

I am new to autocorrelation analysis. I want to test if my sampling points are different when the distance increase between them. So here is my question: When there is no spatial autocorrelation does ...
1
vote
0answers
17 views

Fixed-length time series difference measurement

I have a very specific case of time series comparison. Suppose we have two time series $T_c$ and $T_s$, both of them $512$ discrete points in size. Both of them have been sampled at the same ...
0
votes
0answers
43 views

Calculate similarity of probability vector and TFIDF vector

I want to compute the similarity between documents of two groups of documents. All documents have the same term vector of length 100, but documents in group A are probabilities, whereas documents in ...
1
vote
2answers
74 views

how to measure similarity of two lists of continuous data with different length

I have two lists of continuous data with different length. a) How should I measure (dis)similarity of these two lists? or, as these lists can be formed into histogram, how can I quantify ...
1
vote
0answers
33 views

Cluster analysis of large, multiple answer dataset

I have to analyse data from a marketing study. I will use SPSS. The questionnaire will look like this: Q: Imagine Situation X. Select 1-3 Criteria from the list that best describe your feeling. ...
1
vote
2answers
62 views

Classifying time-series similarity - what variable should I train on?

I have ~10,000 time series, each with 65 time points. I'm interested in classifying each pair of time series as "similar" or "not similar". Here's an example of two similar (left) and not similar time ...
1
vote
0answers
46 views

Similarity between sets with different size

is there a distance measure like jaccard for sets with different sizes? For example A=['a','b','c'] and B=['a','d'] I would like to include the total intersection as well as the order. The ...
1
vote
1answer
60 views

How to find similarity matrix from a confusion matrix?

How to find similarity matrix from a confusion matrix of a multiclass classification? For example in this paper: Exploiting confusion matrices for automatic generation of topic hierarchies and ...
0
votes
0answers
17 views

To what degree does Mahalanobis distance account for correlations of the data?

I understand that Mahalanobis distance is used when your data are correlated (e.g., as with many environmental variables), but just how correlated can the variables be? Should I be screening and ...
0
votes
1answer
33 views

Error when computing Cosine Similarity

I'm trying to compute the cosine similarity of the following vector: 1 1 0 1 2 0 1 1 $ \frac {1*2+1*0+0*1+1*1}{ \sqrt(1^2+1^2+0^2+1^2)\sqrt(2^2+0^2+1^2+1^2)}$ ...
0
votes
1answer
65 views

Order a correlogram

I have 30 days data, and I am plotting correlogram using corrplot (R package). I have an option to order the correlogram with hclust option and hence I obtain plot as shown on right side of below ...
1
vote
0answers
18 views

What does it mean when least squares and instrumental variables give similar results?

What can said when for some data set the result of a least squares regression is extremely close to the result of an instrumental variable regression?
0
votes
1answer
81 views

Alternate distance metrics for two time series

I have time-series data of different houses. Assume it is power consumption data. Now, I want to cluster the houses following similar power consumption pattern utmost. So, the various distance metrics ...
0
votes
0answers
22 views

Similarity measure for short series

I am looking for the best similarity measure between pairs of equal length series. These are concentrations of molecules measured in discrete fractions, the number of fractions is small (5-6). The ...
0
votes
1answer
51 views

Preprocessing survey data for clustering

I want to find 4-10 clusters in survey data with 100 questions answered by 2000 individuals using a technique such as K-means or Gaussian Mixture Models. There is no response variable so the ...
3
votes
1answer
352 views

Implicit feature space of Power Kernel

For the polynomial kernel, $K(x,y) = (x^Ty+c)^d$, the implicit feature space $\phi$ for which $K(x,y) = \phi(x)^T \phi(y)$ is of finite dimension and well known [1][2]. It is also well known that the ...
0
votes
0answers
12 views

How to find similar documents with mixed features

I am working on finding a similar documents problem. I have 3 sets of features ...
0
votes
1answer
65 views

How to find similar document with new vocabulary

I am working on a problem of finding similar documents. I am using a Tf-Idf based vector space model representation of documents and it gives me good results. However when I encounter a document ...
0
votes
0answers
26 views

Multivariate DTW

I am not able to find any resources in case of DTW for multivariate signals. For example, I would like to find DTW similarity between two objects where each object contains following signals: ...
2
votes
2answers
60 views

Proper statistical analysis for similarity of two paired datasets

I am an engineering student and as part of an undergraduate research, I created a device that measures a certain value. However, there is a commercial device that measures that same value. Here's ...
0
votes
2answers
42 views

Cluster users based on file access and application usage

I have access to file access data for employees in my organisation. For each employee I can see what read/write operations they carried out in the last month or so. Per operation, I see a user ID, ...
0
votes
0answers
22 views

Cluster logistic regression datasets

Let's say that I've got a 100 advertising campaigns and for each I'm trying to predict that a user will click on the advertisement or not using logistic regression. So basically I'm creating 100 ...
1
vote
0answers
18 views

Similarity between an item and a set

I have two items $I_0$ and $I_1$ and a set $S$. What I want to achieve is to compare which item (either $I_0$ or $I_1$) is more similar to the set. I can compare both $I_0$ and $I_1$ to every element ...
0
votes
0answers
12 views

Modelling distributions evolving with time

Say we have N probability vectors of dimensionality D describing how a process evolves in N time steps, something like multidimensional time series. What's the best way to model this sequence in ...
1
vote
1answer
67 views

Similarity measures between bimodal distributions

I have to compare two bimodal distributions. So far, I have found the Bimodality Coefficient: $$ BC = \frac{\mu_{3}^{2} + 1}{\mu_{4} + 3\frac{(n-1)^2}{(n-2)(n-3)}}, $$ with $\mu_{3}$ referring to ...
5
votes
1answer
62 views

What is the purpose of row normalization

I understand the reasoning behind column normalization, as it causes features to be weighted equally, even if they are not measured on the same scale - however, often in the nearest neighbour ...
1
vote
1answer
134 views

Similarity / clustering methods for temporal event data

I am looking for methods to calculate a reasonable similarity metric or clustering for my data. I have data in the form of a number of distinct events (lets call them A, B, C, D, ...), and the time ...
0
votes
0answers
42 views

should I count repeated items in jaccard similarity?

I have two lists: A and B where Size(A) = 4 and size(B) = 10. I would like to find the Jaccard similarity between these two lists. Suppose A = {Tom, George, John, Jennifer} and B={Tom, Jessica, ...
0
votes
1answer
54 views

similarity measure [closed]

I have some question concern similarity measure and i need your help (I'am new in statistics) Suppose that we have a matrix $M$ where $M(i, j)$ is the similarity measure between user $i$ and user $j$ ...
3
votes
2answers
46 views

Position-dependent substitution cost matrix for sequence alignment

Many sequence alignment algorithms and those which measure similarity between sequences require specification of a cost matrix defined on the sequence alphabet. However, the cost incurred does not ...
1
vote
1answer
86 views

How to measure similarity between features in different datasets, if possible?

Is it possible to measure the similarity between two different features in different datasets? It could sound like a non-sense question, but it has sense in the context of merging two almost ...
2
votes
2answers
112 views

Similarity measure between groups

Imagine there is a group of people numbered 1-100, each of which have a few numerical attributes e.g. height, weight, age. There is a small sub-group (A) which ...
0
votes
1answer
73 views

Quantification of the extent of periodicity in a time series using fractal analyses

I need metrics to quantify the extent of periodicity between of a time series (for comparison with other time series), considering the time series is almost periodic. By almost periodic I mean: if I ...
1
vote
0answers
32 views

Measure how different are two groups in their answering on 25 ordinal variables

The research context: I'm interested in usage frequency of special interest websites and its predictors. I asked individuals (n=1000) for the usage of 25 different websites in 25 items in a survey ...
0
votes
1answer
33 views

Composite similarity of GitHub projects

As a hobby, I'm building recommendation system which finds related projects on GitHub. I'm computing Jaccard index for each repository, based on users who gave stars: $$J(A, B) = \frac{ \left\vert ...