# Questions tagged [permutation-test]

Statistical tests based on rearrangements of data that are consistent with the null hypothesis.

475 questions
Filter by
Sorted by
Tagged with
1 vote
10 views

### why am i seeing a low F-statistic but a very significant p-value in permanova?

So i am having some numbers which i can't understand...basically i am performing a permanova test using the adonis2 function, from the vegan package, and what i see ...
• 165
15 views

### Going from Individual-Level Significance to Group-Level Significance Analysis Against Chance

I have performed a decoding task (50% theoretical chance level) with data from 36 individuals. 21 out of 36 yielded above chance level decoding accuracies as measured via non-parametric permutation ...
14 views

### Permutation design for repeated measures via how() in RDA

We would like to properly set up the permutation design in anova() testing for our rda(). Study design We chose several areas, ...
• 1
1 vote
13 views

### Does it make sense to "regress out on one variable" during permutation testing?

In our lab we have developed a non parametric permutation test which assesses whether a feature $f:G\to \mathbb{R}$ respects a given symmetric binary relation $A\subset G\times G$. Regard $A$ as being ...
54 views

### Why is the order of factors (order of samples in dataset) affect PERMANOVA results with adonis2?

I am trying to run PERMANOVA tests on multiomics datasets collected from coral samples using adonis2 (v 2.6-5). I found when the order of samples in my dataset change, the pvalues change, sometimes ...
59 views

### Adjusting p-values in multiple regression using permutation test

I have run a multiple regression model using a set of 10 categorical and numerical explanatory variables. This model results in 10 coefficients and their associated p-values. I would like to report ...
• 43
4 views

### Which Statistical test to determine if (Mean) Average Order Values are representative of the population/Stat-Sig?

I am trying to build a process (in python) that measures whether or not the data subsets of an AB test are stat sig. There are 4 total groups. Lets call them Holdout_A, Treament_A, Holdout_B, ...
• 101
1 vote
42 views

### Statistic for significance test comparing transition matrix to null

I have several pitch sequences (mini songs), that look something like this when plotted as a pitch profile (piano roll): I've coded each sequence in terms of its constituent interval sizes, and ...
• 2,112
23 views

### Simpler test equivalent to shuffle tests and (doubly-nested) crossvalidation for the linear, Gaussian case?

My field often uses black-box models to predict a target $y \in \mathbb R^T$ from some covariates $x \in \mathbb R^{T\times N}$. I'm using $T$ to denote the number of samples (presumed independent), ...
• 231
1 vote
46 views

### Permutation Test for mean ratio

I am given three vectors of length 11, that represent blood levels of a hormone measured after placebo, old and new treatment. ...
• 11
101 views

### How do you use numPerms to get the total number of unique permutations for a PERMANOVA?

I am trying to calculate the number of unique permutations for a PERMANOVA using the R adonis2 function but I am getting stuck on defining my permutation scheme. As ...
11 views

### How to use PERMANOVA with non-numeric Y in the formula

I did a MANOVA in R from 10 PCA dimensions I extracted from the result of that analysis with this code: manova_ <- manova(cbind(Dim.1, Dim.2, Dim.3, Dim.4, Dim.5, Dim.6, Dim.7, Dim.8, Dim.9, Dim.10)...
90 views

• 150
25 views

### Computing significance using permutation test in nested cross-validated SVR model

I am running an SVR on multi-dimensional input data (X) to predict an outcome variable (Y). I am using a nested cross-validation approach, with the inner loop using a GridSearch CV to find the optimal ...
• 127
7 views

### How to tally when the permuted test statistic = original test statistic during permutation testing?

I'm doing some permutation testing, and in my case the test statistic is an integer value. So how do I tally the situation where Tpermuted = Toriginal? In that case Tpermuted is neither greater nor ...
• 534
1 vote
89 views

### permutation tests of group differences when data contain repeated measures

I have data from several mice with id id. Each mouse either belongs to either group C or group ...
• 1,159
98 views

### Permutations with repeated measures

Let's say I have data from several mice with id id. Each mouse wandered around two mazes (A and ...
• 1,159
31 views

### Is it possible to pool results from PERMANOVA tests on multiply imputed datasets?

I wondered whether it is possible to pool results from multiple PERMANOVA tests performed on imputed data. In case of ANOVA, methods of how to pool results have been proposed and discussed, e.g. van ...
• 143
1 vote
60 views

### permutation test for mixed effects 3-stage nested design

I have a dataset that uses a 3-stage nested design, and I want to perform an ANOVA. I used lme() from the nlme package in R to fit a linear model, but I found that after doing a shapiro-Wilks test, ...
• 11
32 views

### Variable Importance Methods for Multiple Variables/Features simultaneously

I have fitted a classifier on a number of input variables/features that can be reasonably grouped. I am interested in finding out the importance of sets/groups of variables (as opposed to individual ...
1 vote
34 views

### Independence-violation issue in pairwise measurements: Which test?

Here is another problem in which independence might be an issue (however, the question might be very basic). Suppose, I have a rowing team and I have two boats (B1 and B2, for brevity). I have strong ...
• 11
40 views

### Permutation test to overcome autocorrelation issues?

let's say, I want to track a rowing team's (collective) performance on some task and show that it increases with practice. I have a metric, i.e. the time to measure their performance. Now I would like ...
• 11
33 views

### how to determine significance with permutation t-distribution values

suppose I have the following array of t-distribution values from a permutation test (e.g., linear model which tests, for example, the influence of an adhd score and alcohol consumption on a stress (...
150 views

### Bootstrapping for groups where within each group there are subgroups sampled unevenly

Say I collect two samples from two groups but the sample I collect from each subject within each group is a different size. ...
• 1
1 vote
90 views

### Not sure on how to interpret the p-value for a permutation test

I was running a permutation test on my data. My goal was to test difference in means in two samples, by resampling one of them. I got results like these: And From my understanding, the p-value here ...
• 13
37 views

### How to compare observed versus expected ratio data between two independent, unpaired groups with different sample sizes?

so I'm comparing the expected fitness of double mutant animals (based on independent, additive effects of each corresponding single mutant) to the observed fitness of the double mutant. The data is ...
82 views

### $R^2$ model comparison in test data

I want to compare two models of the form: ...
9 views

### Metric to analyze shuffling on compression

I wish to analyze effect on shuffling on compression. Is there any other metric other than compression ratio and entropy? The metric should coorelate shuffling with a compression ratio.
• 31
95 views

### CLT for t-test with unequal sample size; one group < 30?

CrossValidated has many discussions on how unequal variances are not a practical issue for two-sample t-tests when Welch correction is used and on how normality assumptions do not play a role (in Type-...
• 8,132
14 views

### Statistical test for low treatment sample size, high nb of record and large control group

I am looking to analyse the efficiency of a treatment. The problem is that I have only 3 patients in the treatment group. However: I have thousand of measurement of the value of interest for these 3 ...
• 1,519
1 vote
91 views

### How to conduct a boostrapped or a permutation wilcoxon test (in R)

I have a total of 18 mice, 9 from Healthy group and 9 from Sick group. Since I have a very poor sample size I would like to use boostrapping or permutation test. Is there a way to conduct a ...
• 517
1 vote
105 views

### Calculate a permutation-based p-value for a risk ratio

Summary How do you calculate a two-sided p-value for a risk ratio/relative risk (obtained from a GEE logistic regression via predicting risks with and without a treatment) based on a permutation test, ...
1 vote
35 views

### Is the expected value of a correlation in a randomly permuted sample zero?

Let's say we have two real valued data sets $x$ and $y$, both of length $n$. I do not want to make any further assumptions regarding these data sets. We're interested in their correlation. For testing ...
• 16.8k
1 vote
77 views

### How to calculate p-value from the expected correlation coefficient using randomisation in R?

Here, I want to estimate the p-value using randomization. Tests of correlations involving patterns like Y (a function of X) vs. X involve a shared term and the X term. Here I want to randomize (999 ...
• 583
1 vote
37 views

### Monty Hall Gone Crazy [closed]

Can Someone help me out with these ?Answer along with solution will be much appreciated Q1) Imagine that now host have the following instructions. Put a prize behind a random door. Let the guest guess ...
20 views

### Determine how likely a combination of two counts is compared to null models for both counts

I have two zero-adjusted negative binomial regression models fitted via gamlss not necessarily using the same covariates but at paired locations. ...
• 69
1 vote
236 views

### Permutation testing: Which variable should be shuffled?

When conducting a permutation test, which variable should be shuffled? Let's say I have two variables X and Y, and my test statistic is the correlation coefficient between the two variables. To test ...
• 21
95 views

### Constrained ordination (CAP) pairwise ANOVA

Is there any way to perform pairwise ANOVA test from Constrained ordination for multiple factors in a variable? e.g. I am trying to figure out whether microbial communities among five plant genotypes ...
• 71
1 vote
24 views

### How can I efficiently test whether a subsets of nodes in a DAG are "lined up" more often than expected by random chance?

I have a directed acyclic graph with N nodes, each of which is assigned to one of K groups, with K < N. My hypothesis is that nodes in the same group tend to "line up" along a linear path....
20 views

### Permutation test replications aggregation

Short version: Is it right, and if yes, what is the right way to combine permutation tests applied on n randomly selected subsets of a bigger dataset? I was not able to find a publication that tackles ...
113 views

### Conducting a Permutation test in R

I have a 200 × 102 data matrix of measurements of gene activity in male cancer patients (cases, last 52 columns) and healthy men (controls, first 50 columns). I want to do a permutation test to ...
• 41