My goal is to compute pairwise similarity between 700 regions (i.e., columns of Dataframe #1 described below); hence a 700x700 output matrix.
I have two dataframes I would like to consider in the similarity calculation:
- Dataframe #1 = terms (300 rows) by regions (700 columns), where each element is a numerical continuous value reflecting the importance of a term in a region.
- Dataframe #2 = terms (300 rows) by terms (300 columns) pairwise distance matrix, reflecting the Euclidean distance between the respective terms' word embeddings. This matrix effectively captures how similar in meaning any of the two terms are.
How could I best compute pairwise similarity between 700 regions?
I would like to somehow use Dataframe #2 in the similarity calculation, as some terms are likely redundant in meaning. I do not simply want to run a dimensionality reduction technique on Dataframe #1 to remove redundancy, as I do not believe it will effectively account for semantic redundancy (i.e., the form of redundancy I sought to capture in Dataframe #2).
In the end, I would like to answer the question of how similar any two regions are. I would like to use both dataframes in the calculation. Any recommendations would be appreciated! Thanks!
