4
$\begingroup$

So I am having some numbers which I can't understand...basically I am performing a permanova test using the adonis2 function, from the vegan package, and what I see is a very significant p-value with a very low f-statistic and I was wondering how this is possible.

This is what I have:

Permutation test for adonis under reduced model
Terms added sequentially (first to last)
Permutation: free
Number of permutations: 999

adonis2(formula = transposed_taxa ~ Treatment, data = adonis_meta, permutations = permuts, method = "bray", by = "terms")
          Df SumOfSqs      R2     F Pr(>F)
Treatment  1   0.3506 0.08074 1.581  0.002 **
Residual  18   3.9912 0.91926
Total     19   4.3418 1.00000

I also tried an online F distribution calculator and, if I did the computations correctly (i.e., filled the proper number of degrees of freedom), the calculator gives me a non-significant p-value for the f-statistic I get from the test. Also, I tried the other way around, i.e., I provided the p-value and I got a very high f-statistics. However, I may be completely wrong and/or this calculator does not work for permanova test.

What am I missing here?

EDIT: this code provide the results i am seeing:

# load library
library("vegan")

# set seed
set.seed(131)

# build metadata
meta <- structure(list(treatment = structure(c(1L, 
2L, 2L, 2L, 2L, 1L, 2L, 1L, 1L, 1L, 1L, 2L, 2L, 1L, 1L, 1L, 1L, 
2L, 2L, 2L), levels = c("treat", "ctrl"), class = "factor"), SampleID = c("smpl_001", 
"smpl_002", "smpl_003", "smpl_004", "smpl_005", "smpl_006", "smpl_007", "smpl_008", 
"smpl_009", "smpl_010", "smpl_011", "smpl_012", "smpl_013", "smpl_014", "smpl_015", 
"smpl_016", "smpl_017", "smpl_018", "smpl_019", "smpl_020")), row.names = c("smpl_001", 
"smpl_002", "smpl_003", "smpl_004", "smpl_005", "smpl_006", "smpl_007", "smpl_008", 
"smpl_009", "smpl_010", "smpl_011", "smpl_012", "smpl_013", "smpl_014", "smpl_015", 
"smpl_016", "smpl_017", "smpl_018", "smpl_019", "smpl_020"), class = "data.frame")

# build distance matrix with bray curtis metric
dist_mat <- structure(c(0.724637600797927, 0.810151211442847, 0.677040935047858, 
0.684990809226583, 0.773841077947218, 0.730965541771736, 0.721636429281786, 
0.670062302186762, 0.735219755374301, 0.726420817041101, 0.713890852756624, 
0.672481163496018, 0.770155244262034, 0.736284810905502, 0.725613337053673, 
0.729027437249588, 0.684174177335026, 0.731597842534973, 0.744984703810319, 
0.624016277905062, 0.716705596289758, 0.642368044368606, 0.660041089097984, 
0.683430554390529, 0.637694222022575, 0.696747540033531, 0.678073194220625, 
0.606862380345106, 0.57055508520052, 0.551073859282405, 0.685299854417393, 
0.576578503650073, 0.70888030875411, 0.753358012196228, 0.559701576529528, 
0.628139583570118, 0.616139151118536, 0.673897628859827, 0.688365913838432, 
0.600801291804287, 0.653870227670454, 0.713403074119784, 0.622562333387564, 
0.628100195801016, 0.638223638807926, 0.621583052281211, 0.639684018803079, 
0.658541917209453, 0.62143045183579, 0.711481887009044, 0.753101948589314, 
0.652118731984756, 0.669405500109936, 0.631233748354645, 0.681137745516244, 
0.75990843296923, 0.704545253550062, 0.733485020719486, 0.70159943122252, 
0.687665942056175, 0.71194111729798, 0.687556422198947, 0.672450715128387, 
0.729058498800823, 0.777450338280718, 0.724047236746885, 0.693695657946689, 
0.649024860632527, 0.743978052380142, 0.690306665100079, 0.66738824968164, 
0.666456720997265, 0.736939480231912, 0.701398808286334, 0.666061916019718, 
0.673314770321561, 0.678582532793769, 0.601722977395924, 0.694594001795866, 
0.666715778630456, 0.733905602972688, 0.708928297345911, 0.660957906863241, 
0.688380488856927, 0.66749857777161, 0.638562693219007, 0.709099622247953, 
0.661582073250648, 0.633361258020319, 0.64395516014329, 0.649660296892919, 
0.693266403675858, 0.619593981263679, 0.610426150945898, 0.680254103567928, 
0.704351725342505, 0.655961485324122, 0.682236566670637, 0.60440942570617, 
0.717559929520578, 0.678396363326736, 0.662793617648454, 0.657884773482801, 
0.611508752657517, 0.695249293387878, 0.641413832130993, 0.631289777942896, 
0.76873495257605, 0.7214801270069, 0.648605405437042, 0.638675714854417, 
0.690476980816459, 0.624196684143274, 0.716828273978603, 0.620623296876388, 
0.734305024372298, 0.696968360773382, 0.731531545670681, 0.590046390437714, 
0.656457660565246, 0.767672873743218, 0.662583670754548, 0.741384141808377, 
0.632153877463622, 0.686203048119573, 0.681849943901682, 0.679030715444911, 
0.695081943961127, 0.670973436524669, 0.655707226050489, 0.592390953713221, 
0.719296118412277, 0.681242617504174, 0.709011062976595, 0.67783231497305, 
0.628507609160785, 0.698992117288368, 0.692043273990139, 0.635338402719872, 
0.657103639492005, 0.716464174187625, 0.615559649585827, 0.695082938920572, 
0.667734702508218, 0.616496859490065, 0.642779258886188, 0.653646025092285, 
0.631693957514602, 0.611457903279869, 0.688517163734204, 0.737259708862597, 
0.603583398496329, 0.656242818753164, 0.597835522574495, 0.594058163092095, 
0.670135094778774, 0.621802926897363, 0.744151393706153, 0.723831207989047, 
0.563513462547226, 0.628881872570669, 0.62794706034554, 0.676684493053249, 
0.622600780389654, 0.698874934198522, 0.7223868940598, 0.552933805235236, 
0.664090363014779, 0.611138112893922, 0.596969123444625, 0.73702614785897, 
0.66350703692813, 0.648073310354135, 0.611886898022537, 0.698582583703285, 
0.683497738003825, 0.753238008973899, 0.637095874502176, 0.625844086528647, 
0.629241237689765, 0.732814733043977, 0.641962575290711, 0.769763636147583, 
0.711868946785738, 0.701186509994886, 0.689603853801978, 0.766601551308245, 
0.634042239970803, 0.624152094184768, 0.701782607184278), maxdist = 1, Size = 20L, Labels = c("smpl_001", 
"smpl_002", "smpl_003", "smpl_004", "smpl_005", "smpl_006", "smpl_007", "smpl_008", 
"smpl_009", "smpl_010", "smpl_011", "smpl_012", "smpl_013", "smpl_014", "smpl_015", 
"smpl_016", "smpl_017", "smpl_018", "smpl_019", "smpl_020"), Diag = FALSE, Upper = FALSE, method = "bray", class = "dist")

# run test
print(adonis2(dist_mat~treatment, data=meta, permutations=999, by="terms"))
$\endgroup$
4
  • 1
    $\begingroup$ You apparently have quite a small dataset. Can you edit your post to include the data (by pasting in the output of dput(adonis_meta)) and the code you ran? $\endgroup$ Mar 28, 2023 at 11:13
  • 1
    $\begingroup$ the dataset comprises 20 samples (10 for each treatment) and ~4.5k taxa since these are ecology data. so, yes it may be possible that the number of samples is small but i have no idea how big the average dataset looks like. and i am afraid i won't be able to upload the dataset. $\endgroup$
    – gabt
    Mar 28, 2023 at 12:35
  • 1
    $\begingroup$ you could post the dissimilarity matrix which should (i think?) be sufficient for this test. $\endgroup$ Mar 28, 2023 at 15:08
  • 1
    $\begingroup$ @GeorgeSavva, yep, indeed it is. i did it, thank you! $\endgroup$
    – gabt
    Mar 29, 2023 at 8:11

1 Answer 1

6
$\begingroup$

There is a reason why the tests are based on permutation instead of nominal values of F-statistic. The reason is that nominal values of F-statistic are not correct in this setting, but we must find the empirical distribution under null model from permutations.

You can have a look at the permutation distribution of F using function permustats to extract permutation values (randomized values) and densityplot and other support functions to display those values, or densityplot(permustats(<yourmodel>)).

$\endgroup$
3
  • 1
    $\begingroup$ ok, so the F statistics that i see in the result is this pseudo-F that is used to compute the p value using the formula mentioned in the wikipedia page (en.wikipedia.org/wiki/Permutational_analysis_of_variance). so, the pvalue is giving me the likelihood of observing a F statistic that is bigger (more extreme) than the one adonis2 is returning? what does this tells me about the data, though? $\endgroup$
    – gabt
    Mar 28, 2023 at 13:25
  • 1
    $\begingroup$ The interpretation is the same as with normal F-values in frequentist testing and it tells you the same thing about data as usual frequentist P-values: probability of your data given Null model. $\endgroup$ Mar 29, 2023 at 8:39
  • $\begingroup$ alright, thank you! $\endgroup$
    – gabt
    Mar 29, 2023 at 12:27

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.