Stack Exchange Network

Stack Exchange network consists of 174 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Measures of closeness between distributions, clusterings, data sets or other objects.

0
votes
0answers
15 views

Extracting Product Info From Free Form Text [on hold]

I got a list of promotion texts and I want to match products in these texts to products in my database. For example: Promotion Text: Oreo 6 pack 50% off ...
0
votes
1answer
20 views

Similarity index between two texts Ask Question

I'm trying to compare two vectors in a small NLP project using Python. Code doesn't make any difference since I'm using scikit-learn, but my doubts are about my calculations. I have a query vector ...
0
votes
0answers
10 views

optimal dissimilarity measure for time series count data

I have time series of three variables x,y,z. All are based on count data. Is there an optimal dissimilarity metric to obtain differences between respective pairs of x,y,z? Is it appropiate to use ...
0
votes
1answer
22 views

Similarity between 2 profiles (observations). Is it possible to generate a % similarity?

I have multiple profiles for 10 different people. Each person has been measured for 5 different continuous variables of different magnitudes. So my dataframe is 10x5 where each row represents a person ...
0
votes
0answers
8 views

Somers' D deteriorating for more granular rankings

Background I have 30 observations and I want to compare how much experts and model agree with each other. For Model and Expert I have slots which are $\in [1,4]$ and ranking which are $\in [1,30]$. ...
0
votes
0answers
19 views

How to find Mahalanobis distance between an observation and a population with mixed data?

I have a dataset(let's call it as 'D') with multiple continuous, nominal and ordinal variables, as follows: continuous: Total sales as well as sales figures of some products by customers nominal: ...
1
vote
0answers
23 views

Similarities / suggestions of multi-attribute entities

Dataset: I have 200 000 entites, these have zero or more of 1000 attributes (commonly 1-20 of them) The attibute relation is weighted 1-3 Its stored in a relational database modeles as Entity, ...
2
votes
0answers
32 views

Similarity between Train and Test data sets

I have multiclass classification dataset and I am using Deep nets for the classification task. To explain the problem, let's assume that I have 5 classes to classify. No matter what I try, be it ...
-2
votes
1answer
37 views

Find similar images in a dataset without labels [closed]

I have a set of grayscale images, some of them are transformed of the other images. For example in 10 images, image 2 is the same as image 8 but rotated, and image 4 is the same as image 7 but ...
0
votes
0answers
55 views

Get similarity of two 2D probability distribution maps

I have two 2D probability distribution maps (hexbin maps), showing the probability of something showing up in a specific hexagon. So it's discrete, and each hexagon has a number as probability. One ...
0
votes
1answer
39 views

Clustering visited vertex in a graph

I would like to know can anyone introduce me an algorithm to cluster multiple sequence of visited vertices in a graph. For example assume your graph has 6 nodes. following might be 3 different ...
0
votes
0answers
18 views

Type of Recommendation engine algorithm to use

I'm planning to build a recommendation engine for a market place. The market place has User1,User2,,,,,,,UserN (with new users joining all the time). Each user has city and state information. Market ...
-2
votes
1answer
17 views

R Find similiar combinations of entries in vectors

I have a problem where I have no idea how to solve it. I have many character vectors which entries are mostly containing 1-4 words. Here´s some sample data: ...
1
vote
0answers
17 views

Predicting categories based on a similarity matrix

I am looking for some help in organising some analyses. I will describe what I am trying to do with a fictional example and then talk about some of the things I've thought about already. Example I ...
0
votes
2answers
49 views

Similarity function for machine learning training examples

I have a bunch of training examples (which are time series segments) that are used to train an algorithm. For any new example that is now presented to the algorithm for classification, i want to the ...
0
votes
0answers
15 views

Quantifying similarity of two objects in a population on certain characteristics

I have a set of objects (these are businesses) and for these businesses I have data on a number of characteristics. To keep this simple, we will consider these to be number of staff number of ...
0
votes
0answers
12 views

Positional similarity

What is a good positional similarity measure? Bottom line - words, which are next to each other should be ranked higher (no semantic or fuzzy matching needed). I want to use this in addition to TF/IDF-...
0
votes
0answers
30 views

Similarity of two sets of points

PROBLEM I have one set of 10 points, $X = \{(x_1,y_1),\,\dots,\,(x_{10},y_{10})\}$ and two sets of 3 points each, $A = \{ (a_1,b_1),\, (a_2,b_2),\, (a_3,b_3) \}$ and $C = \{(c_1,d_1),\,(c_2,d_2),\,(...
0
votes
0answers
34 views

Evaluate the similarity of distributions that are a function of a continuous parameter

Consider a continuous bound independent variable $X$ (for context, the altitude angle measured on a telescope) for which one can measure zero or more values of a continuous dependent variable $Y(X_i)_{...
0
votes
0answers
23 views

Compare two set

I want to compare two sets of values with a ranking. I want to have more of a similarity coefficient. My goal is to take into account both aspects: the similarity on the ranking and the similarity on ...
1
vote
1answer
15 views

similarity distance when weight should change

I am trying to find a similarity measure for a very specific problem. I have a list of cities that have some characteritisques. For example bilingual school, firemen stations, etc. These cities are ...
0
votes
0answers
11 views

Is it possible to compare species compositions using some kind of analysis: anosim, adonis or PERMANOVA but using presence/absence of data?

Is it possible to compare species compositions between three groupss using some kind of analysis: anosim, adonis or PERMANOVA but using presence/absence data?
0
votes
0answers
45 views

On the right dissimilarity matrix for discrete cluster analysis

Currently I'm having some trouble on deciding the right dissimilarity matrix for my data. After reading some previously asked questions here, I'm even left more confused. So, here is the deal. My data ...
1
vote
0answers
18 views

Distance vs Dissimilarity measure

I am reading up on distance and dissimilarity measures for my class on natural language processing and could not understand this slide. Why does the dissimilarity measure not satisfy item 3 ? What ...
0
votes
0answers
28 views

Using some kind of similarity measure to predict target variable from historical data

Let me first describe the data I have: I have aggregated CAN bus data (1 second windows, from the original 50 ms values) from 6 forklifts, and I also have measurements for these 6 trucks which ...
1
vote
0answers
30 views

“Fuzzy” but simple and elegant similarity measure for time series

I am looking for a similarity measure between series which exploits the time dimension (rather than just its "mapping" aspect), and was unsatisfied with everything I found. First an example: assume I ...
0
votes
1answer
96 views

Best similarity measure for binary wine data

I'm creating a dataset of wine grape varieties and their associated flavors/aromas. Here's a schematic of the data: ...
2
votes
1answer
39 views

What is a “Nominal Type Histogram” and “Shuffling Invariant Property”?

I'm reading this paper about comparing histograms ["Comprehensive Survey on Distance/Similarity Measures between Probability Density Functions" by Sung-Hyuk Cha] that writes in the introduction: ...
11
votes
2answers
211 views

Does Mercer's theorem work in reverse?

A colleague has a function $s$ and for our purposes it is a black-box. The function measures the similarity $s(a,b)$ of two objects. We know for sure that $s$ has these properties: The similarity ...
0
votes
0answers
14 views

Similarities between two networks

I am studying the similarities and differences between two research areas in terms of their descriptors (key words). I have two corpus corresponding to documents derived from both research areas. By ...
0
votes
0answers
58 views

Regression of pairwise trait similarity on similarity in personal attributes

I want to relate how simlar people are on their metabolite levels, to their similarity on personal attributes. Let's say I have measured metabolite levels for 500 metabolites in 1000 subjects. For ...
0
votes
0answers
29 views

similarity between position frequency matrices

I need to check the similarity between a set of position frequency matrices (finally to see if there is significant difference between 2 groups of 8 matrices). Simplified example of two matrices are ...
0
votes
0answers
16 views

Denominator in pairwise similarities in symmetric SNE

I am trying to understand how symmetric SNE works while reading the paper of Van der Maaten about t-SNE. However, I am a bit confused about the denominator in the following expression for the ...
0
votes
0answers
24 views

What would be a good approach to find similar rows between datasets?

I have 4 different datasets with information about people. Examples of features in these datasets are region of residence, job, education, and age. These datasets represent annual catchments over 4 ...
-1
votes
2answers
109 views

Methods for evaluation of clustering

I have labeled data set (with only 2 classes) and I'm trying different clustering algorithms with different variations of similarity measures (which creates different distance matrixes that I give as ...
0
votes
1answer
21 views

Test Similarity of two models

I have two models that I've trained using different ML algorithms. I want to compare how well they match each other as a test of whether they are generalising the training set or simply memorising it. ...
0
votes
0answers
144 views

Similarity between documents in LDA “word” vectors space?

Using LDA we can convert a bag-of-words document into a vector of topics $\theta$ to incorporate statistical relations between words, then document similarity can be measured in the "topic" space by ...
2
votes
1answer
168 views

Are time series motifs and the Matrix profile algorithm a good fit for my problem?

I have huge multivariate time series to analyze (Terabytes of data) and I need fast, scalable algorithms for mainly two tasks: finding similar patterns among time series. For example, imagine I ...
0
votes
0answers
49 views

Baroni-Urbani and Buser Overlap Index - How to summarize?

I have a dataset that consists of multiple sets of 3 paired tissues. For example, for a given mouse, I have a tumor sample, a lung sample, and a blood sample all from that same individual, and then I ...
1
vote
1answer
226 views

User-user suggestions with collaborative filtering (recommendation system)

I have a binary matrix N x N where both rows and columns represent users of a website. If matix[i,j] = 1 it means that ...
0
votes
0answers
20 views

Is there a metric to measure “degree of deviation” between two strings?

I'm trying to figure out a metric that can estimate the "degree of deviation" between two strings. For example, if I have two strings such as "abcdfhg" and "aaeefhf", I want to calculate how much the ...
0
votes
0answers
42 views

Clustering huge amount of vectors

I have a huge amount of vectors (about 2 millions). Vectors may be different by size (length ranges from 1 to 86K), each element of a vector is a number and the elements are sorted. (E.g. v_1=[1,3,4,5,...
0
votes
1answer
157 views

Comparing similarity of two distributions

I'm building a simulation that outputs data in distributions and I need a metric to know how similar they are to ground-truth data. Specifically, I output things like this, which is a distribution of ...
1
vote
1answer
79 views

Map distance between two time series to a probability

We have several time series, which are basiclly chunks of numeric values. We use Dynamic Time Warping to calculate the distance between these time series. This is working well and gives us some ...
1
vote
0answers
38 views

Transforming dissimilarity matrix for use as weights in network

I'm trying to identify communities or clusters in a network by optimizing the modularity function. The network is a fully connected network, with undirected edges that have weights based on a pairwise ...
0
votes
0answers
47 views

Levenshtein distance “proper” range for fuzzy matching

I am hoping to use Levenshtein distance concept for fuzzy matching in SQL joins between two tables on t1.first_name||last_name = t2.first_name||last_name. Is ...
0
votes
0answers
21 views

Similarity of two attributes with different number of categories

I'm a little lost in the web und struggle to find the right keywords. I have about 1000 items for two categorical attributes, but they have different number of categories. E.g.: Let attribute A be ...
0
votes
0answers
20 views

Clustering Suggestion for highly imbalanced medical data with binary and numerical variables

I have a highly imbalanced medical data set with approx 4k binary cohorts(variables) and 100 numerical variables. I want to separate patients into different clusters based on similarity. I tried using ...
0
votes
1answer
10 views

Given a series of sequences of events at certain times, how can I calculate the degree of similarity / dissimilarity?

I'm analysing a series of feature analyses for live music, where identified segments or key changes are associated with a given timestamp, i.e: ...
0
votes
0answers
136 views

Measure similar to Jaccard index but comparing more than two groups at a time

I am interested in looking at the amount of sharing between several samples (each represented as a vector of character strings), and to do pairwise comparisions, I have been using Jaccard indices. Is ...