0
$\begingroup$

Related: Average distance in distance matrix

I'm looking for some way to compare euclidean distance matrices. The matrices I need to compare will have constant number of rows but varying number of columns. My goal is to find to best or nearest match (pick the best matrix). Ideally there is a function that takes such a matrix as an argument and outputs a single value that I can sort, but that may be too much to hope for. I'm looking for something for robust than min or max distance. Can I do better than average distance?

Above stream, I have two data frames or tables. The first is constant, and the second is a sequence of subsets or filters based on a 'code' column. For each pair of tables, I'm computing the euclidean distance matrix (which will have various dimensions since at each iteration I'm dynamically finding all columns in common. Unfortunately the columns of the two master tables only overlap by around 25% of a few thousand). Feel free to suggest other approaches besides euclidean distance or distance matrices in general, if my approach is paradigmatically flawed.

Statistically, I'm worried about the implication of having different sets of columns for each comparison. This results from generation of dummy variables, the underlying representations of which vary. Consequently, for the (admittedly arbitrary) cases I've examined, most of the columns are entirely zero's, since there is little overlap between the factors that are turned into dummies.

|cite|improve this question
$\endgroup$
0
$\begingroup$

Maybe you could try using simple matching koeficient. Why do you need distance, for clustering or something else?

|cite|improve this answer
$\endgroup$
  • $\begingroup$ thanks for the suggestion, I will check out SMC. good question - it's kind of like pseudo-clustering. I've spent a ton of time on KMeans clustering, but it is too hard for my limited computational resources. Here, I manually selected one group (by external label) from one dataset; and want to iterate through a second dataset's external-label-groupings to find the best match (by manual inspection, I also know the second dataset's external label that should correspond with the first's). The hope is that I can scale this up to the hundreds of datasets I have, then move onto another primary label $\endgroup$ – Bjorks number one fan Jan 11 '18 at 18:14
  • $\begingroup$ it looks like SMC is inappropriate since only some of my variables are binary; I have several columns of doubles/floats $\endgroup$ – Bjorks number one fan Jan 11 '18 at 18:26
  • $\begingroup$ yes but you can transform variable to binary easily. Let say you have two rows, and value of one variable for one row (observation) is 5, and another row is 8, now you transfor this for first row 1,1,1,1,1,0,0,0 and other will be 1,1,1,1,1,1,1,1 and now you can look for distance. But as i noticeted you have multile variable in both tables this maybe is not good approach. Are this variable of a same type? $\endgroup$ – explorer Jan 12 '18 at 7:18
  • $\begingroup$ I've never heard of such a technique, but I guess it makes sense in terms of distance. My main concern is that a) the table already has thousands of columns due to dummy variables (from strings), and b) since many of my doubles are thousands, this would explode the size of the matrix. my laptop can barely handle the current size of tables (in fact it cannot handle some of the tables, it freezes and I have to force shutdown). do you know how to reduce dimensionality here or do you think the SMC statistic is worth all the trouble? currently i'm using avg of avg distances between observations $\endgroup$ – Bjorks number one fan Jan 12 '18 at 14:01

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.