Should I use Hellinger transformed species (abundance) data for NMDS if this is what I used for RDA ordination? I am conducting both constrained and unconstrained analyses on the same species abundance data. For the constrained ordination, I ran the RDA (redundancy analysis) on log x+1 and hellinger transformed species abundance data. For the unconstrained ordination, I am using NMDS. I initially did the nmds using raw (non transformed) species data:
metaMDS(SPdata, distance = 'bray',k=2, trymax=100)

This automatically applies Wisconsin double standardization.
However, I just saw an example on YouTube where they also used the hellinger transformed data for the nmds.
If I am hoping to look at both the RDA and NMDS results for comparison, should I also be using the hellinger transformed data for the nmds? like I did for the RDA.
I just tried the NMDS with the hellinger transformed species matrix and it does give slightly different results.
I am using the vegan package in R for both ordinations.
I feel like this is a very basic question, but I have a limited stats background and haven't found a satisfying answer online.
Thank you!
 A: This is a good question.
To answer this let's think about the two methods (NMDS and db-RDA) a litle deeper.
NMDS is an "indirect gradient" analysis approach using ordination.  The axes are imaginary, that is, they do not represent real distances.  Unlike metric ordination approaches (MDS/PCoA, PCA), which maximize the variance between ordination site scores in the ordination space, NMDS attempts to preserve the pairwise dissimilarity between sites in a low-dimensional ordination space. It does this using species ranks (not abundance) and their variation among sites. Thus, distances between two points are relative and representative of the maximal compositional variation among sites but are not real distances (they are imaginary).  Any distance measure can be used to construct the NMDS (Bray-Curtis is advisable if you have abundance data, Sorensen if you have presence/absence)
db-RDA on the other hand is a "direct gradient" analysis approach that extracts real (i.e., known distance) variation in the response variable (community dissimilarity among sites) which can be attributed to a redundant set (i.e., collected at the same sites) of environmental predictors, using multiple linear regression. db-RDA does not use ranks but actual distances (like PCA). Hellinger transformation is warranted in many un-even community datasets, to better linearize the distances among sites (and reduce the effect of dominant species in the site dissimilarity). see Ecological meaningful transformations by Legendre: http://adn.biol.umontreal.ca/~numericalecology/Reprints/Legendre_&_Gallagher.pdf.  However, a distance measure is still used (on top of Hellinger-transformation if used).  You could use the same distance measure in both analyses for consistency (i.e., Bray-Curtis is advisable if you have abundance data, Sorensen if you have presence/absence), but because the distances are known quantities for db-RDA and imaginary quantities for NMDS they would only be qualitatively (not directly)  comparable across ordination spaces.
So, the two methods call for different treatment of the community matrix prior to their application, based on the math/assumptions of the ordination approach / fitting the linear model for db-RDA. you want to make sure you use the correct / most-justified
