Questions tagged [permutation-test]

Statistical tests based on rearrangements of data that are consistent with the null hypothesis.

Filter by
Sorted by
Tagged with
0
votes
0answers
23 views

The way PERMANOVA handling continuous explanatory variable

I have read some tutorials which explain the mechanism behind PERMANOVA, e.g., enter link description here. Based on the tutorials, my understanding is PERMANOVA can only handle categorical ...
0
votes
0answers
24 views

Is it correct to use permutation testing to find the significance of a clustering solution?

I am performing exploratory clustering with k-means in a multidimensional (82x18) dataset. The algorithm that I am using tests several number of clusters and uses several goodness of fit metrics to ...
1
vote
0answers
23 views

Restricted Permutations for Repeated Measures Nested Plot PERMANOVA

Apologies, for this might be somewhat convoluted. I have an experiment examining community-level differences among treatments and are using PERMANOVA along with other multivariate methods to do this. ...
1
vote
0answers
34 views

What to do if an exact permutation test is infeasible

For permutation tests, it is often infeasible to enumerate all permutations. There are two approaches to solve this. The first uses a random number of permutations instead of all permutation (this is ...
1
vote
0answers
33 views

Permutation Tests: How to Interpret a Non-zero-centered Null Distribution

In a "textbook" experiment (e.g., a single treatment is randomly given to a subset of independent units, the SUTVA holds, etc.), my understanding is that a permutation test of the experiment should ...
0
votes
0answers
16 views

Should I include all the covariates in a PERMANOVA test (R)?

I have a matrix with abundance ( df_for_abundance) from gut microbiota and a matrix with metadata (26 variables; meta). I would like to calculate the effect of clinical covariates on the composition ...
1
vote
0answers
41 views

Wilcoxon signed-rank test on cross-validated values

Many neuroscience papers aim to estimate the significance of classification accuracy measures (AUC-ROC for example): can one classify such behaviors better than chance with such neuronal activities? ...
0
votes
0answers
20 views

Valid significance testing of coefficients when performing model selection

I want to know how I can validly perform permutation (or bootstrap) significance testing of model coefficients when I am also using the same dataset to perform model selection. My model is not the ...
0
votes
0answers
16 views

lunar phase influence on animal abudance in stomach content of predator species

I have a set of data presenting the abundance of prey in the stomach of one predator species. each predator was identified with an ID number and date was recorded among other info. The date was used ...
1
vote
0answers
36 views

Can we perforn a permutation test with zero-inflated data?

In a collaborative problem-solving task, I would like to check if the mean number of communicative exchanges emitted by a person is different following whether they shared an emotion just before or ...
0
votes
0answers
36 views

Interpreting results of resemblance-based permutation methods: PERMANOVA and PERMDISP?

I am working with an invertebrate data set (i.e., counts of individuals per invertebrate order, captured by pitfall trap) and am exploring trends in community composition in relation to environmental ...
0
votes
0answers
12 views

Hypothesis tests via permutation importance on validation data

A model agnostic way to measure variable importance of a variable $x$ in a statistical model (originately proposed by Breiman) is as follows: Select a scoring function $\mu$ to measure model ...
2
votes
1answer
45 views

Examples for exact tests which are not permutation tests

Wikipedia states (without citation): Sometimes permutation test is used as a synonym for exact test, but although all permutation tests are exact tests, not all exact tests are permutation tests. ...
0
votes
1answer
21 views

Number of all sample divisions in permutation test

I have a question with regard to the number of permutations in a permutation test. Assume I want to do a two group independent t-test. In the permutation variant of the t-test, I would repeatedly ...
0
votes
1answer
23 views

Do all non-parametric statistical tests involve bootstrapping, simulation, or permutation test?

All of the non-parametric tests I learned about involved bootstrapping, simulation of large samples of some random variable, or permutation tests. Are there non-parametric tests without simulation or ...
0
votes
0answers
21 views

Is it normal to have symmetric centroids' coordinates in a NMDS (with two factor levels)?

I have performed a NMDS (using the metaMDS() function of the vegan package) on a set of plant communities. The analysis gave a satisfactory stress value (ca. 0.08) as well as an okay-looking shepard's ...
0
votes
0answers
32 views

Bootstrapping of data, in R, for improving the homogeneity of variance

I have a data collected on five point likert scale; aims at studying, the level of preference individuals gives to the different factors, (such as Tax Benefits, Maturity period, Rate of Returns ...) ...
2
votes
0answers
82 views

How permutation based FDR control methods take into account the correlation structure?

There are a number of permutations-based False Discovery Rate (FDR) control methods, such as SAM [1], dChip [2], Ge et al [3], Meinshausen [4], and Efron [5]. It's been said that in these methods, ...
0
votes
0answers
90 views

PERMANOVA test and its assumptions

I have collected data based on a 5 point Likert scale (very low, low, neutral, high, very high) on factors considered by individuals before making investment decisions. There are five factors (D.Vs) ...
0
votes
0answers
23 views

Permutation Manova

I have collected the data based on 5 point likert scale(very low, low, neutral, high,very high) on the factor considered by individual before making investment decision. There are five factors (D.Vs) ...
4
votes
3answers
138 views

Is there a formula to calculate all possible unique permutations of n elements over p positions?

Is there a formula to calculate all possible unique permutations of n elements over p positions? Please imagine the following ...
0
votes
1answer
37 views

Sampling from the null distribution for a t-test

I am trying to apply the Dietterich 5x2 cv t-test to test whether one statistical model is better than another on a particular data set. In order to apply this test, one must assume that under the ...
0
votes
0answers
21 views

Did I appropriately run a permanova on an additive model with nested factors in R?

Because I don't have a statistician readily at my disposal and I am still fairly new to multivariate statistics, I hope maybe one of you out there could maybe check whether I did an appropriate ...
0
votes
1answer
32 views

How can I generate a completely balanced combination

Attaching a code that generates unique combinations of 8 letters but, it still lacks a condition wherein all letters should have equal counts per column. For my code, basically, letter a to h should ...
0
votes
1answer
74 views

In VEGAN ADONIS, what does "method='bray' do when I pass in a distance matrix I already have (UNIFRAC)?

Suppose I have a distance matrix created by applying the weighted unifrac measure. For instance, using a phyloseq object: ...
0
votes
0answers
4 views

Example of Permutations of Objects which are Alike [duplicate]

Can you give an example that let me visualize the formula of the number of permutation of n objects in which k1 are alike, k2 are alike, etc. n!/k1!×k2!×...×kn! where k1! + k2! + ... + kn = n Also ...
0
votes
0answers
29 views

bootstrap testing - p value with subtraction and replication (Python)

from std.disclaimer import "Dammit Jim, I'm a programmer, not a statistician." Edited to try to make the terminology more correct and the question more clear. Mea ...
1
vote
2answers
57 views

Bootstrap for CIs and Permutation Resampling for Hypothesis Test?

In various scattered places online and StackExchange I've read that bootstrap resampling is more appropriate for calculating confidence intervals while permutation resampling is more appropriate for ...
3
votes
2answers
113 views

Two-tailed one-sample permutation test

I want to test the significance of the classification accuracy of a cross-validated binary classifier. I performed a permutation test to determine whether the true classification accuracy is higher ...
1
vote
0answers
43 views

Using a permutation test to test for differences in the median between two groups when distributions of the variable of interest are not equal

in the following, I will define (1) the hypothesis I wanted to test, (2) the method I used, and (3) the problems that I feel are present. My dataset consists of measures of one dependent variable for ...
0
votes
0answers
74 views

Is two group comparison test combinig rarefaction and permutation usefull and correct?

I wish to know if the analytical procedure using R environment described below is statistically coherent to enable interpretations and conclusions. I looking for evaluations and opinions about the ...
0
votes
0answers
9 views

Compute effect size from simulated NULL distribution

I computed a empirical p-value (permutation test) where I compare my real value to simulated values (N=100,000). In the end I've a distribution like this (vertical line = my real value). For the p-...
2
votes
0answers
36 views

Is this a valid non-parametric approach for computing false discovery rate (FDR)?

Summarizing the method in the reference below as I understand it: For each of $N$ "gene sets" (where a gene set represents a single null hypothesis significance test), compute a test statistic $T_i$. ...
3
votes
1answer
93 views

How did these researchers determine the confidence interval of the AUROC using resampling but without retraining the model?

In this Nature article backed by Google, the investigators develop then externally validate a deep learning model for predicting lung cancer using CT scans. In their internal validation results, we ...
1
vote
0answers
18 views

Is a group structure required for randomization tests?

In randomization tests (for example in Testing Statistical Hypothesis Theorem 15.2.1 Lehmann and Romano) it is required that we have a group of transformations $\mathbf{G}$ on which $X$ and $gX$ have ...
1
vote
0answers
30 views

Difference between three randomization methods for permutation test

Source: http://www.lock5stat.com/StatKey/randomization_1_quant_1_cat/randomization_1_quant_1_cat.html There are three randomization methods for conducting the permutation test which is a non-...
1
vote
0answers
115 views

Comparison of classification accuracy with permutation tests [closed]

My dataset contains 4 balanced classes (A, B, C and D). By carrying out two independent binary classifications, I want to compare the classification accuracy of (A vs. B) with that of (C vs. D). ...
0
votes
0answers
22 views

How to calculate the significance of slope and y-intercept using a permutation test of pairwise fst x distance

I have two variables, 1. scaled pairwise Fst (measure of genetic differentiation between two sites) 2. river kilometers between the two sites. The values are not independent (all of the sites are ...
0
votes
1answer
23 views

Do randomization tests for difference in means assume independent groups?

I performed a randomization test and found that the means of groups A and B are significantly different. I am trying to prove that group B is "special", that there is something interesting going on ...
0
votes
1answer
66 views

Randomization test vs. t-test vs. others?

I am trying to find out if the means of two groups are different. I have performed a randomization test and found significant results but I would like to confirm it with some other test. I looked into ...
0
votes
0answers
46 views

Permutation test for accuracy in multiclass discrimination

It is well known that to assess the significance (i.e. difference from chance) of the performance of a classification analysis one can use a permutation test. In a two-classes case, the procedure is ...
0
votes
0answers
39 views

Correction for multiple testing of permutation p-values

I simultaneously performed multiple statistical tests on my samples under investigation. For each test i obtained p-value that i had to correct for ...
0
votes
0answers
16 views

Re-randomization for cluster randomized trial, implications for primary analysis

I am hoping the masses might have some insight or a couple papers to point to on the issue I am facing. My research group conducts cluster-randomized trials where we assign treatment by cluster (...
0
votes
0answers
11 views

Is it mathematically sound to use binary independent variables in a permutational ANOVA since it does not assume multivariate normality?

I am trying to determine if it is mathematically sound to use binary independent variables in a permutational ANOVA since it does not assume multivariate normality. I was considering Canonical ...
1
vote
0answers
52 views

How to calculate permutation test p-value when resulting computation of permutations is a matrix of test statistics

I am working with 2 matrices consisting of matrix 1 (xrows, ycols) and matrix 2 (yrows, zcols). My resulting computation is a matrices multiplication which results in a matrix of (xrows, zcols). I ...
0
votes
0answers
34 views

Difference in results of permutation tests through coin::oneway.test and resample::permutationTest2

I'm trying to run a same permutation test for the null hypothesis of no difference between means with the coin and resample R ...
0
votes
0answers
50 views

RDA axes significance

I performed a RDA with RDA function from vegan library. I tested model significance with permutations and I also teste axes significance. Results indicate that the model is significant (p=.005). From ...
1
vote
0answers
50 views

How to calculate a permutation confidence interval?

Mara Tableman, Minh Nguyen, and Michael D. Ernst note The idea behind a permutation confidence interval is well known and fairly straightforward (Ernst 2004). The confidence interval is simply the ...
2
votes
3answers
94 views

Can we calculate a confidence interval using a permutation testing approach?

Tim Hesterberg's "What Teachers Should Know about the Bootstrap" is excellent. On page 77 he writes: "We cannot use permutation testing to obtain confidence intervals." Is that correct? If so, can ...
0
votes
0answers
53 views

Statistical test for dependent data- comparing groups of pairs

I have a question about statistical tests for dependent data. This will sound a bit strange/extremely naive but pls bear with me- here is my problem formulation: I have 500 individuals, and I'm ...

1 2 3 4 5 8