The tag has no wiki summary.

learn more… | top users | synonyms

1
vote
1answer
17 views

Similarity Coefficient and Geochemical Correlation

I have a set of around 160 samples from different sediment bodies, or stratigraphic units, which I am attempting to correlate using geochemical analysis of 25 elements for each sample, and a ...
2
votes
0answers
26 views

What is a good technique for grouping objects based on binary or dichotomous traits?

I have a set of objects each of which has a list of traits. Data on the traits is binary: an object has a trait or does not. The number of objects that I have is moderately greater than the number ...
0
votes
0answers
29 views

Converting a similarity to a dissimilarity [duplicate]

I am working on a clustering problem for which I have to manually choose the number of clusters. I have a visualization tool that helps me decide whether the clusters are good. In order to ...
4
votes
0answers
89 views

Compute Shannon entropy between every row of a large, sparse matrix

I have a sparse, binary matrix of user (rows) and items (columns). Each element of this matrix is either 0 or 1: ...
1
vote
0answers
29 views

Cosine similarity for a group of vectors

I am starting to work in R with the following case: 35 countries have scores between [0,1] in 120 dimensions and I need to show that there are differences in the way their scores distribute across ...
0
votes
2answers
65 views

Similarity measure between two variables

I want to find the similarity factor (some numerical value) between two variables. Example: row 1: 5.1, 3.5, 1.4, 0.2 rows 2: 4.9, 3.1, 1.5, 0.1 How to find similarity factor between these two ...
1
vote
1answer
42 views

Determining similar users from hierarchical clustering

I use hierarchical clustering to cluster users which are similar to each other based on a Jaccard coefficient. I have now coded a solution to extract similar users based on hierarchical clustering: ...
2
votes
0answers
22 views

How to calculate the similarity of two corpora (each of which contains a set of documents)?

I have two corpora for example, each of which contains a set of different documents, and each document are already represented as a vector of words in a certain way. The two corpora are small, only ...
0
votes
0answers
38 views

Learning from distance matrices

I have a task to build multi-task classification model based on distance (similarity) matrix only. They are already precalculated and no changes here can be applied. Can you, please, recommend me ...
0
votes
0answers
33 views

Dataset similarity measurement

Suppose: For each input data, I could obtain true result as well as experiment result from my algorithm, one example is The true result is A = {a, b, c}, ...
0
votes
2answers
45 views

Similarity measure between multiple distributions

To compare distributions, it is common to use box blots. I'm looking for a similarity measure that calculates whether distributions are the similar or not. Ideally, given that e.g. four ...
1
vote
1answer
61 views

Clarification needed about min/sim hashing + LSH

I have a reasonable understanding of the technique to detect similar documents consisting in first computing their minhash signatures (from their shingles, or n-grams), and then use an LSH-based ...
2
votes
0answers
186 views

How to get Bhattacharyya Distance in Excel (or Matlab, or R)

I have several histograms which I would like to compare to one reference histogram to see which one is the most similar to the reference in terms of the shape of the distribution. Kologorov-Smirnov ...
0
votes
2answers
90 views

How similar are my 2 data sets?

I am kind of stuck with an easy question: I have two data sets with experimental data. The data sets do not have the same size. I would like to show that these data sets are possibly coming from the ...
0
votes
0answers
22 views

How a problem of calculating a probability of two differently represented objects being the same is called?

I'm solving a task of matching (i.e., asserting that different "accounts" represent the same object) social network's profiles of users with their representation in a database of firms founders. The ...
3
votes
4answers
137 views

How do I deal with large data similarity computation?

I have lot of records like this: M is about 10 million and N is about 100K. Now I want to apply collaborative filtering on ...
1
vote
1answer
73 views

similarity measures with missing values

How to handle missing values when computing similarity (or distances)? (I have binary feature values and do use the simple matching coefficient, but I feel that the answer to this question may be more ...
0
votes
0answers
12 views

Comparing two items with various parameters of different units

Not a big stats guy, but I dabble in programming and frequent stackoverflow and figured this would be a good place to turn. Here's what I'm looking for. I have a set of food items. Each has a number ...
3
votes
0answers
44 views

Kernel interpretation of Manhattan distance / Lp norms

Euclidean distance corresponds to the linear kernel similarity via: $$d_\text{Euclidean}(x,y)^2=\sum_i (x_i-y_i)^2=\sum_i x_i^2 + y_i^2 - 2 x_i y_i = ...
0
votes
2answers
46 views

Negative values distance model between 2 sets

I have two sets, $A$ and $B$, containing string values. I am trying to model the relationship between sets $A$ and $B$ by computing the following distance: $d = \frac{ \mid A \mid + \mid B \mid- \mid ...
2
votes
0answers
82 views

Comparing facial curvature of two human faces

I need help determining if what I'm doing is sufficient. I have extracted the "curvature" from two human faces (essentially the curve that goes from one ear, down the face to the chin, and back up to ...
0
votes
1answer
90 views

coding survey data for cosine similarity and euclidean distance?

I want to know how to code survey data such that a similarity function can be applied on it. Say I want to use cosine similarity. All the search results and QA I've found while in my search deal only ...
0
votes
0answers
26 views

Finding similarities using Wavelet transform

I have a time serie and I want to find similarities in it. For the first step I have calculated Haar-wavelet coefficients for this time serie, and now I don't know exactly how should I continue ...
1
vote
1answer
69 views

Combining similarity scores

I have a list of m x n similarity score matrix, something like ...
0
votes
0answers
57 views

How to analyse environmental temperature datasets

I need some advice regarding my statistical analyses with SPSS. I have a dataset of temperatures recorded automatically every 30 minutes in three different habitat types for the same time frame. I ...
0
votes
0answers
15 views

Most similar combination of partitions to the rest of group

I have a number of sample sequences of data that I want to compare with another large number of sequences of data. These samples are partitioned in between and I want to see which combination of ...
0
votes
0answers
41 views

Chi-Square Difference in Similarity Matrix

I am attempting to implement an algorithm and to construct a similarity matrix looking at the pairwise similarity between N distributions. The instructions are that each element in the similarity ...
0
votes
0answers
90 views

Computing Jensen-Shannon Divergence between discrete and continuous distribution

Is it possible to compute the Jensen-Shannon divergence between a discrete and a continuous probability distribution, e.g. between a standard normal distribution and a distribution taking the values ...
0
votes
1answer
75 views

What methods exist to measure how similar a set of sampled distributions are to each other?

I have a set of distributions that are sampled. How do I measure their similarity between themselves or as a group/set? So, given a finite domain for a sampled function$F_j(x)$, with $x_i \in ...
-1
votes
1answer
77 views

Cluster centroids visualization

I've some data that I've clustered and calculate the similitude within my cluster centroids to spot which class centroids are close to which ones. My goal was to visualize this as a map of points. I ...
0
votes
0answers
59 views

How to explain LSA results

I used an LSA (Latent Semantic Analysis) code in python to find the similarity of two documents. The results of the code are the following, but I cannot understand what is the score of the similarity. ...
1
vote
0answers
44 views

Appropriate distance measures

I am working with hyperspectral image data in R, so I have subset an image to a region of 5000 pixels, each containing a vector 254 bands in length. I would like to cluster this data in order to try ...
1
vote
1answer
198 views

recommenderlab, why does the similarity function return only 0's and 1's? [closed]

In a matrix with user ratings I want to calculate similarities between the first ten users and the rest of the users. I use the method "jaccard" for the similarity calculation in recommenderlab. My ...
2
votes
1answer
241 views

Kullback-Leibler divergence of two normal distributions

I was recently trying to find a way to compute the KL-divergence between 2 populations that are normally distributed using the mean and variance of each population. But I found several different ...
0
votes
0answers
175 views

Clustering text with python

I asked on stackoverflow but they suggest me to move here for better answers. I copy paste the question. I decide to play a little with similarities and clustering text. I have already create the ...
1
vote
0answers
72 views

Computing similarity in LSA

I have a question regarding Latent Semantic Analysis (LSA) introduced in http://lsa.colorado.edu/papers/JASIS.lsi.90.pdf‎ . On page 14, schemas for calculating similarities between different types of ...
3
votes
2answers
82 views

Similarity theory: Testing whether dimensions are separable or integral

Note: I'm not referring to linear separability. I've found the interesting comment in Edelman, Shahbazi: "Renewing the respect for similarity" that for integral dimensions, Euclidean distance is ...
0
votes
0answers
18 views

Statistical similarity on sequences

What are the methods that can be used for evaluating the similarity of the sequences? Lets say that we have three sequences; I want to be able to say that "statistically" the sequence number 1 and the ...
2
votes
0answers
41 views

Creating statistically similar sequences

I have to run a test on a program, with some statistical tools to make sure its results are statistically acceptable. This program will take sequences of data information (a vector of integers or ...
0
votes
0answers
96 views

How to compute similarity between two Dirichlet distributions?

I have a set of distributions and I would like to compute their similarity. Each distribution is a distribution over distributions of topics. If I have just a distribution over some topics, I could ...
3
votes
1answer
396 views

Measurement of similarity for hierarchical clustering trees

I am performing a number of hierarchical clusters on a dataframe of patient records (e.g. similar to http://www.biomedcentral.com/1471-2105/5/126/figure/F1?highres=y) I am experimenting with ...
1
vote
1answer
64 views

Similarity metric not based on statistical correlation

I'm building a simple recommender algorithm and as a prerequisite to calculating a prediction, I need to assign a similarity metric between each two pair of users. The next step being calculating a ...
3
votes
2answers
198 views

Clustering of points based on vector feature similarities in R

I have as an input a number of points that I need to partition into clusters. Each point has a number of features that are ideally to be used to find the similarity between each point and the others. ...
2
votes
0answers
57 views

Combining evidence from different sources

I am computing similarities between words using similarity measures based on wordnet. I have obtained similarity measures for two words using three different methods - JNC (Jiang and Conrath), BNP ...
5
votes
1answer
483 views

Pattern recognition techniques in spatial or spatio-temporal data?

I am working with weather forecasters and have access to historical climatology data. Given current weather conditions in an area of interest (i.e. the current "map"), we want to try to find the most ...
5
votes
1answer
1k views

Similarity Coefficients for binary data: Why choose Jaccard over Russell and Rao?

From Encyclopedia of Statistical Sciences I understand that given $p$ dichotomous (binary: 1=present; 0=absent) attributes (variables), we can form a contingency table for any two objects i and j of a ...
1
vote
2answers
126 views

Similarity between objects based on tags (binary features)

I have five millions of objects each of them having one or more tags. How do I compute statistically sound similarity score between each pair of the objects taking into account that: There are 100 ...
0
votes
2answers
158 views

Time Series Similarity : Differing Lengths with R

I am experimenting with creating a distance matrix between time series for clustering and similarity searching. The main reference I am using is for the Similarity procedure in SAS (Paper). I would ...
0
votes
0answers
120 views

Distance Metrics For Binary Vectors

I have vectors of same length consisting of 1 and 0. I am trying to find out how similar they are. So far I am using hamming distance that I calculate sum of one vector then sum of second vector and ...
0
votes
1answer
121 views

Finding the similarity between two functions

I am a first-year grad student in Computer Science, and I need some help with a problem that I think is statistically oriented. I have taken a statistics course, but it was abysmal and I haven't had ...