Questions tagged [permutation-test]
Statistical tests based on rearrangements of data that are consistent with the null hypothesis.
475
questions
1
vote
1
answer
10
views
why am i seeing a low F-statistic but a very significant p-value in permanova?
So i am having some numbers which i can't understand...basically i am performing a permanova test using the adonis2 function, from the vegan package, and what i see ...
- 165
0
votes
0
answers
15
views
Going from Individual-Level Significance to Group-Level Significance Analysis Against Chance
I have performed a decoding task (50% theoretical chance level) with data from 36 individuals. 21 out of 36 yielded above chance level decoding accuracies as measured via non-parametric permutation ...
- 1
0
votes
0
answers
14
views
Permutation design for repeated measures via how() in RDA
We would like to properly set up the permutation design in anova() testing for our rda().
Study design
We chose several areas, ...
- 1
1
vote
0
answers
13
views
Does it make sense to "regress out on one variable" during permutation testing?
In our lab we have developed a non parametric permutation test which assesses whether a feature $f:G\to \mathbb{R}$ respects a given symmetric binary relation $A\subset G\times G$.
Regard $A$ as being ...
0
votes
1
answer
54
views
Why is the order of factors (order of samples in dataset) affect PERMANOVA results with adonis2?
I am trying to run PERMANOVA tests on multiomics datasets collected from coral samples using adonis2 (v 2.6-5). I found when the order of samples in my dataset change, the pvalues change, sometimes ...
4
votes
1
answer
59
views
Adjusting p-values in multiple regression using permutation test
I have run a multiple regression model using a set of 10 categorical and numerical explanatory variables. This model results in 10 coefficients and their associated p-values. I would like to report ...
- 43
0
votes
0
answers
4
views
Which Statistical test to determine if (Mean) Average Order Values are representative of the population/Stat-Sig?
I am trying to build a process (in python) that measures whether or not the data subsets of an AB test are stat sig. There are 4 total groups. Lets call them Holdout_A, Treament_A, Holdout_B, ...
- 101
1
vote
0
answers
42
views
Statistic for significance test comparing transition matrix to null
I have several pitch sequences (mini songs), that look something like this when plotted as a pitch profile (piano roll):
I've coded each sequence in terms of its constituent interval sizes, and ...
- 2,112
0
votes
0
answers
23
views
Simpler test equivalent to shuffle tests and (doubly-nested) crossvalidation for the linear, Gaussian case?
My field often uses black-box models to predict a target $y \in \mathbb R^T$ from some covariates $x \in \mathbb R^{T\times N}$.
I'm using $T$ to denote the number of samples (presumed independent), ...
- 231
1
vote
0
answers
46
views
Permutation Test for mean ratio
I am given three vectors of length 11, that represent blood levels of a
hormone measured after placebo, old and new treatment.
...
- 11
2
votes
1
answer
101
views
How do you use numPerms to get the total number of unique permutations for a PERMANOVA?
I am trying to calculate the number of unique permutations for a PERMANOVA using the R adonis2 function but I am getting stuck on defining my permutation scheme.
As ...
- 41
0
votes
0
answers
11
views
How to use PERMANOVA with non-numeric Y in the formula
I did a MANOVA in R from 10 PCA dimensions I extracted from the result of that analysis with this code: manova_ <- manova(cbind(Dim.1, Dim.2, Dim.3, Dim.4, Dim.5, Dim.6, Dim.7, Dim.8, Dim.9, Dim.10)...
- 21
2
votes
1
answer
90
views
How do I find the number of unique permutations for a PERMANOVA with an unbalanced design?
From the PRIMER manual (p. 28 about Monte Carlo permutations; http://updates.primer-e.com/primer7/manuals/PERMANOVA+_manual.pdf), the number of unique permutations for a PERMANOVA with $a$ groups and $...
- 41
2
votes
1
answer
119
views
What kind of null hypothesis can a permutation test consider?
From what I have seen, permutation tests are always used to compare the means (or median) of two different groups in the experiment. Even Wikipedia says
A permutation test involves two or more ...
- 107
1
vote
0
answers
63
views
Can we apply a permutation test to compare two percentage distributions?
Any help here will be greatly appreciated!
Suppose we have two groups, one composed of ET's from Mars and another from ET's from Jupiter (assuming there are ETs there).
Then, these groups of ETs meet ...
4
votes
1
answer
185
views
Can I use a permutation test on timeseries data?
What I am trying to do:
I am currently doing analysis on neuronal calcium imaging data.
In particular, I have two things:
A time series that represents the amount of calcium within a neuron
A boolean ...
- 43
7
votes
1
answer
4k
views
Why does R say 'cannot compute exact p-values with ties' when I can do it with pen and paper?
Suppose I have two sets of three numbers: $x_1, x_2, x_3$ and $y_1, y_2, y_3$ and I want to test the Null hypothesis that they are drawn from the same distribution using the Wilcoxon-Mann-Whitney test....
- 802
1
vote
0
answers
16
views
How do I assess the significance of classification accuracy, after determining hyperparameters with GridSearchCV and testing with train_test_split?
So I've recently learned that if I use GridSearchCV to select the best hyperparameters AND evaluate model performance, this can lead to an optimistically biased measure of performance due to ...
- 31
1
vote
1
answer
163
views
Small sample size: Permutation test or Wilcoxon test?
What do you think about using a permutation test each time the sample size is too small to meet the the assumptions of normality and homogeneity of variance etc ? Why do we use a non parametric ...
- 517
2
votes
1
answer
100
views
Paired data and Bootstrapping : Can I use a boostrap method on paired data
I have patients (20 total) that took both "treatment1" and "treatment2" and I am analyzing the change of a response variable, I use paired wilcoxon test to analyse this data.
But I ...
- 517
1
vote
0
answers
14
views
Permutation Testing Using only Source Samples
Short Background on Permutation Testing
Suppose I have two sets of samples $P$ and $Q$ drawn iid from distributions $\mathcal{P}$ and $\mathcal{Q}$ over $X$.
I also have access to a test function $T: ...
- 150
0
votes
0
answers
25
views
Computing significance using permutation test in nested cross-validated SVR model
I am running an SVR on multi-dimensional input data (X) to predict an outcome variable (Y). I am using a nested cross-validation approach, with the inner loop using a GridSearch CV to find the optimal ...
- 127
0
votes
0
answers
7
views
How to tally when the permuted test statistic = original test statistic during permutation testing?
I'm doing some permutation testing, and in my case the test statistic is an integer value. So how do I tally the situation where Tpermuted = Toriginal? In that case Tpermuted is neither greater nor ...
- 534
1
vote
1
answer
89
views
permutation tests of group differences when data contain repeated measures
I have data from several mice with id id. Each mouse either belongs to either group C or group ...
- 1,159
3
votes
1
answer
98
views
Permutations with repeated measures
Let's say I have data from several mice with id id. Each mouse wandered around two mazes (A and ...
- 1,159
0
votes
0
answers
31
views
Is it possible to pool results from PERMANOVA tests on multiply imputed datasets?
I wondered whether it is possible to pool results from multiple PERMANOVA tests performed on imputed data.
In case of ANOVA, methods of how to pool results have been proposed and discussed, e.g. van ...
- 143
1
vote
0
answers
60
views
permutation test for mixed effects 3-stage nested design
I have a dataset that uses a 3-stage nested design, and I want to perform an ANOVA. I used lme() from the nlme package in R to fit a linear model, but I found that after doing a shapiro-Wilks test, ...
- 11
0
votes
0
answers
32
views
Variable Importance Methods for Multiple Variables/Features simultaneously
I have fitted a classifier on a number of input variables/features that can be reasonably grouped. I am interested in finding out the importance of sets/groups of variables (as opposed to individual ...
- 1
1
vote
1
answer
34
views
Independence-violation issue in pairwise measurements: Which test?
Here is another problem in which independence might be an issue (however, the question might be very basic).
Suppose, I have a rowing team and I have two boats (B1 and B2, for brevity). I have strong ...
- 11
0
votes
0
answers
40
views
Permutation test to overcome autocorrelation issues?
let's say, I want to track a rowing team's (collective) performance on some task and show that it increases with practice. I have a metric, i.e. the time to measure their performance.
Now I would like ...
- 11
0
votes
0
answers
33
views
how to determine significance with permutation t-distribution values
suppose I have the following array of t-distribution values from a permutation test (e.g., linear model which tests, for example, the influence of an adhd score and alcohol consumption on a stress (...
- 1
0
votes
0
answers
150
views
Bootstrapping for groups where within each group there are subgroups sampled unevenly
Say I collect two samples from two groups but the sample I collect from each subject within each group is a different size.
...
- 1
1
vote
1
answer
90
views
Not sure on how to interpret the p-value for a permutation test
I was running a permutation test on my data. My goal was to test difference in means in two samples, by resampling one of them.
I got results like these:
And
From my understanding, the p-value here ...
- 13
0
votes
0
answers
37
views
How to compare observed versus expected ratio data between two independent, unpaired groups with different sample sizes?
so I'm comparing the expected fitness of double mutant animals (based on independent, additive effects of each corresponding single mutant) to the observed fitness of the double mutant. The data is ...
0
votes
1
answer
82
views
0
votes
0
answers
9
views
Metric to analyze shuffling on compression
I wish to analyze effect on shuffling on compression. Is there any other metric other than compression ratio and entropy? The metric should coorelate shuffling with a compression ratio.
- 31
2
votes
1
answer
95
views
CLT for t-test with unequal sample size; one group < 30?
CrossValidated has many discussions on how unequal variances are not a practical issue for two-sample t-tests when Welch correction is used and on how normality assumptions do not play a role (in Type-...
- 8,132
0
votes
0
answers
14
views
Statistical test for low treatment sample size, high nb of record and large control group
I am looking to analyse the efficiency of a treatment.
The problem is that I have only 3 patients in the treatment group. However:
I have thousand of measurement of the value of interest for these 3 ...
- 1,519
1
vote
0
answers
91
views
How to conduct a boostrapped or a permutation wilcoxon test (in R)
I have a total of 18 mice, 9 from Healthy group and 9 from Sick group. Since I have a very poor sample size I would like to use boostrapping or permutation test.
Is there a way to conduct a ...
- 517
1
vote
1
answer
105
views
Calculate a permutation-based p-value for a risk ratio
Summary
How do you calculate a two-sided p-value for a risk ratio/relative risk (obtained from a GEE logistic regression via predicting risks with and without a treatment) based on a permutation test, ...
- 32
1
vote
1
answer
35
views
Is the expected value of a correlation in a randomly permuted sample zero?
Let's say we have two real valued data sets $x$ and $y$, both of length $n$. I do not want to make any further assumptions regarding these data sets. We're interested in their correlation. For testing ...
- 16.8k
1
vote
1
answer
77
views
How to calculate p-value from the expected correlation coefficient using randomisation in R?
Here, I want to estimate the p-value using randomization. Tests of correlations involving patterns like Y (a function of X) vs. X involve a shared term and the X term. Here I want to randomize (999 ...
- 583
1
vote
0
answers
37
views
Monty Hall Gone Crazy [closed]
Can Someone help me out with these ?Answer along with solution will be much appreciated
Q1) Imagine that now host have the following instructions. Put a prize behind a random door. Let the guest guess ...
- 11
0
votes
0
answers
20
views
Determine how likely a combination of two counts is compared to null models for both counts
I have two zero-adjusted negative binomial regression models fitted via gamlss not necessarily using the same covariates but at paired locations.
...
- 69
1
vote
1
answer
236
views
Permutation testing: Which variable should be shuffled?
When conducting a permutation test, which variable should be shuffled?
Let's say I have two variables X and Y, and my test statistic is the correlation coefficient between the two variables. To test ...
- 21
4
votes
2
answers
95
views
Constrained ordination (CAP) pairwise ANOVA
Is there any way to perform pairwise ANOVA test from Constrained ordination for multiple factors in a variable?
e.g. I am trying to figure out whether microbial communities among five plant genotypes ...
- 71
1
vote
0
answers
24
views
How can I efficiently test whether a subsets of nodes in a DAG are "lined up" more often than expected by random chance?
I have a directed acyclic graph with N nodes, each of which is assigned to one of K groups, with K < N. My hypothesis is that nodes in the same group tend to "line up" along a linear path....
- 333
0
votes
0
answers
20
views
Permutation test replications aggregation
Short version:
Is it right, and if yes, what is the right way to combine permutation tests applied on n randomly selected subsets of a bigger dataset? I was not able to find a publication that tackles ...
0
votes
0
answers
113
views
Conducting a Permutation test in R
I have a 200 × 102 data matrix of measurements of gene activity in male cancer patients (cases,
last 52 columns) and healthy men (controls, first 50 columns). I want to do a permutation test to ...
- 41
0
votes
0
answers
346
views
Permutation test involving multiple treatment groups where the test statistic is the precision of a single group
In a setting where we conduct multiple experiments in parallel, I am struggling to figure out whether reporting the significance of the precision of the measurements from a single experiment is a ...
- 143