0
$\begingroup$

I work in the field of linguistics, and my current project involves lots of distance matrices generated from language data, which measure the distance and similarity of dialects. Concretely, my distance matrices range between 0 and 1, where 0 represents no distance and 1 represents maximal distance between dialects. Now, I am wondering if there exists statistical significance tests or something like that, wherein we can test if dialect A and dialect B are significantly farther apart than B and C? Alternatively, is there a customary threshold, say 0.5, whereby distances > 0.5 indicates dialects are more different than similar? For instance, consider the distances between CMM_Press and CTM_Press on one hand, and CMM_Other and CTM_Other on the other hand in the distance matrices below.

distance matrix 1

distance matrix 2

distance matrix 3

distance matrix 4

In the four distance matrices above, I am especially interested in the distances of the following pairs:

CMM_Press and CTM_Press: 0.2, 0.19, 0.5, 0.4;

CMM_Other and CTM_Other: 0.6, 0.41, 0.4, 0.69.

Is there significance tests with which I can test if, say, CMM_Other and CTM_Other are significantly farther apart than CMM_Press and CTM_Press?

To facilitate the answering of the questions, you can find the dataset, a R markdown file containing the distance matrices and the scripts for the analysis in this OSF link. The method and the R package that fuel the analysis is quite novel, so perhaps this article could also help explaining how the distance matrices are calculated. Szmrecsanyi B, Grafmiller J and Rosseel L (2019) Variation-Based Distance and Similarity Modeling: A Case Study in World Englishes. Front. Artif. Intell. 2:23. doi: 10.3389/frai.2019.00

In addition, I would like to know if is there exists a good reference on how to interpret distance matrices (e.g., from an ecology point of view where the Mantel test was invented).

Added new information:

Here is how I calculated the distance matrices. I first fit nine mixed-effects logistic regression models and conditional random forest models to nine linguistic datasets with the same set of predictors and the same response variable. For the mixed regression models I use lme4 package in R, and for the random forest models I use party package in R. Then, I check the extent to which across the nine models (a) predictor significance ratings overlap (i.e. significant vs. non-significant), (b) regression coefficients are similar, and (c) variable importance rankings (based on random forest models) are similar. Euclidean-based distance matrices are calculated based on the assessments above and the distance values are scaled to an interval between 0 (minimum distance) and 1 (maximum distance). For instance, in the first distance matrix based on (a), namely predictor significance, the distance value between "Century19th" and "CMM_Other" is 0, which means the two models (i.e. the two dialects in my dataset) completely overlap with each other in terms of the predictor significance. Per (a), (b), and (c), I calculate a distance matrix. And finally I created a fused distance matrix by merging the distance matrices (1), (2), and (3) using the R function analogue::fuse, hence the forth distance matrix. I skipped some of the technicalities due to limited space, and you are kindly referred to the article I mentioned above for more details.

$\endgroup$
8
  • 1
    $\begingroup$ I don't think this can be answered without you telling us a) how distance matrices are calculated, and b) how come you have four of them? $\endgroup$
    – Eoin
    Oct 17, 2022 at 8:36
  • $\begingroup$ New information added. Thanks for the advice! $\endgroup$ Oct 17, 2022 at 9:19
  • $\begingroup$ "Alternatively, is there a customary threshold, say 0.5, whereby distances > 0.5 indicates dialects are more different than similar?" This question generally needs to be answered by the subject matter expert rather than by statisticians. Whether dialects are "more different than similar" is a question about dialects really. Reading your distance description (in the question here) I'm afraid it will be very hard to interpret the distance values properly here, as distance is assessed in a quite indirect way, i.e., hard to relate to the original data. $\endgroup$ Oct 17, 2022 at 13:05
  • $\begingroup$ "In addition, I would like to know if is there exists a good reference on how to interpret distance matrices" - I think that the distance matrix should be defined in such a way that it can be well interpreted by you in the first place. Obviously interpreting distances totally depends on what the distances mean, and this is determined by their definition, so there won't be any general reference. (It may be that you mean something other than interpreting the actual values by "interpreting the matrix", but this is then not clear to me.) $\endgroup$ Oct 17, 2022 at 13:22
  • $\begingroup$ Without saying anything about the rest of your definition, which is hard to get my head around without knowledge of the data underlying these models, I don't think the "significant vs. insignificant" part of the distance is a good idea. Whether something is significant or insignificant is decided by a rather arbitrary threshold, p=0.06 is arguably much closer to p=0.04 than p=0.001 is, and "the difference between significance and insignificance is itself not significant", see here: stat.columbia.edu/~gelman/research/published/signif4.pdf $\endgroup$ Oct 17, 2022 at 13:33

1 Answer 1

0
$\begingroup$

I believe that what you have is actually a dissimilarity measure, instead of distance. Distances don't have an upper limit, (di)similarities do. That have been said, what you want is either a simple Mantel test or a PROCRUSTES/PROTEST analysis. Those are available in the R package vegan

$\endgroup$
2
  • $\begingroup$ Thanks for this. But aren't the Mantel test and PROTEST analysis for measuring the overlap/similarity between distance matrices? My question is not if (and how) these distance matrices are generally similar or not, but are some of the individual distance (or dissimilarity, as you pointed out) values in those matrices significantly farther apart from others in the same matrices. $\endgroup$ Oct 18, 2022 at 13:23
  • $\begingroup$ A PROCRUSTES/PROTEST or a Mantel test are testing for a congruence and correlation, respectively between two matrices. If you're only interested in the structure of a single matrix maybe what you want is a constrained ordination, like PCA. I don't know how to test if a single value is different from others in a single matrix $\endgroup$ Oct 19, 2022 at 12:51

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.