Questions tagged [permutation-test]
Statistical tests based on rearrangements of data that are consistent with the null hypothesis.
503
questions
0
votes
0
answers
8
views
Looking for Composition Differences in a Multi-Year Dataset
I have a multi-year dataset in which I am interested in a genotype (which I can code as a factor) over time for a particular species. I am interested in if there are significant differences in ...
1
vote
1
answer
23
views
Correct permutation design for envfit (nested study design)?
I have the following study design: 8 locations, 5 forest types/"stands" (Plots) per location (D,DB,B,FB,F). In every plot I sampled spiders and identified them. I did an 3 dimensional NMDS ...
- 59
1
vote
1
answer
32
views
Nested studydesign - Necessary to restrict permutations in betadisper?
I have the following study design: 8 locations, 5 forest types (Plots) per location (D,DB,B,FB,F). In every plot I sampled spiders and identified them. I did an NMDS with Morisita-Horn index.
...
- 59
1
vote
0
answers
25
views
Is the permutation test possible even when the randomization scheme is a restricted randomization?
Imagine an experimental setting where the participants are assigned randomly to some treatment groups. However, a form of restricted randomization is used (known by the experimenter).
After the ...
- 51
2
votes
1
answer
26
views
Permutest or TukeyHSD for group-wise comparison after significant Anova of betadisper?
I want to perform a PERMANOVA with my community-data of spider assembladges, sampled in different forest stands. Before applying PERMANOVA I wanted to find out if there is a difference in the ...
- 59
2
votes
1
answer
78
views
How do I perform a permutation test on a machine learning model to obtain a p-value for its performance?
this is kinda of the same question of this previous post. But since there's no reply, and I'm having a hard time to find some answers, I'd like to ask it again.
I'm training a regression model (SVM ...
- 73
2
votes
1
answer
61
views
mean of medians consistently differs from median
Disclaimer: my first question here.
Background of the problem:
I want to compare two distributions (n1 ~ 500 and n2 ~ 700), not normal and with different variances, but roughly unimodal. I decided to ...
- 21
1
vote
1
answer
33
views
Estimating Confidence in Feature Rankings from Multiple Experiments with Non-Normal Data
Hello dear Cross Validated Community,
I am a new doctoral student in bioinformatics, and I am working on a project involving multiple experiments, each generating a single numerical result for each of ...
0
votes
0
answers
8
views
Using the jackknife method in permutation tests
We aim to employ a permutation test to evaluate the significance of the relationship between two datasets using Sparse Canonical Correlation Analysis (SCCA) for a specific level of sparsity. To ...
- 1
1
vote
0
answers
63
views
Plotting the mean difference of two different sample sizes, with the effect size in R
I have a simple data frame of approximately 1750 participants. But have taken the first 11 participants to form the example. From the all_groups variable, I have 4 ...
Nic
0
votes
0
answers
11
views
Proportion testing from pooled single cell/nucleus samples
I have a snRNAseq experiment with a 2x2 layout (wt vs transgenic, drug vs vehicle; Wt-V, Wt-D, Tg-V, Tg-D). I have 6 mice per group, but for cost reasons we pooled pairs of samples in each group, for ...
- 153
1
vote
0
answers
30
views
Can p-values from permutation tests be interpreted as a measure of effect size between group?
I'm conducting a research project involving gene expression analysis.
We've developed a method to calculate a score from a given set of genes, A.
To evaluate the ...
- 113
1
vote
0
answers
17
views
Is a permutation test valid in comparing a fraud block policy false positive rate across two different months?
I'm looking to conduct a counterfactual experiment in an e-commerce fraud context. When a fraud policy blocks a transaction you necessarily will never know if it was truly fraudulent. I'm trying to ...
- 11
0
votes
0
answers
49
views
Algorithm for conducting Westfall & Young permutation test under Multiple Hypothesis Testing given a list of raw(unadjusted) p values
I am currently trying to set up a python program to implement Westfall & Young (1993)'s minP permutation procedure using Python. The computationally efficient algorithm I was able to find by ...
- 1
1
vote
0
answers
35
views
Repeated measures microbiome data with baseline measurement: comparison of two treatments
I am interested in any advice I can get on different ways I might analyze multivariate ecological community (fecal sample microbiome) data that come from a "before after control impact" ...
- 295
0
votes
0
answers
15
views
Necessary assumptions for a permutation test of conditional independence
Consider a case in which you want to know if two variables (X, Y) are independent conditional given a set (C) of other variables. A recent paper (Shah, R. D., and J. Peters. 2020. The hardness of ...
- 111
1
vote
0
answers
57
views
What is the connection between permutation tests and rank tests?
I was trying to understand the null hypothesis of the Wilcoxon rank sum test and thought that the null distribution of the rank sum could be obtained by a permutation test/Monte Carlo test.
If this ...
- 1,616
0
votes
1
answer
63
views
Why is the order of samples in my metadata changing my PERMANOVA results with adonis?
I've trying to preform a PERMANOVA using adonis2 on a dissimilarity matrix. My code looks like this.
...
0
votes
0
answers
15
views
Sanity checking custom cluster extent permutation test of time series data correctly
I am currently developing a cluster extent permutation test on time series data. For that I wanted to sanity test to see whether there are any biases with my test when it is ran on null (i.e., no ...
- 11
0
votes
0
answers
40
views
correct formula for permutation test [duplicate]
in order I need do:
measuring the targeted variable on the original network and save it Shuffling my original data (correlation matrix) - randomly (100x) create the graph measuring the targeted ...
0
votes
1
answer
94
views
Test for statistically significant difference: Wilcoxon vs Median Permutation
I have a dataset that shows measured 'perceived quality'(dependent variable), by two different sizes in the context of 4 different intensity levels (200 values for perceived quality in each category, ...
- 3
6
votes
1
answer
212
views
Permutation test for exponential null hypothesis: really bad?
Having found nice formulas for testing the null hypothesis under exponentially-distributed samples, I wanted to see how well permutation tests could do the job. And the answer, assuming no mistakes, ...
- 1,176
1
vote
1
answer
70
views
When and why is a likelihood ratio preferable to a difference as a test statistic?
We want to know whether two sample sets {x} and {y} were drawn from the same distribution. The null hypothesis $H_0$ is that they are. As statisticians we test the hypothesis by calculating the p-...
- 1,176
1
vote
0
answers
37
views
Is a permutation test with max statistics appropriate to account for multiple comparisons when searching for correlations between two questionnaires?
I'm trying to understand if the max statistic method for permutation test is appropriate to use and how i should apply it in my situation:
I have a sample of 38 individuals who performed a task and we ...
5
votes
4
answers
625
views
What is the information in an exact p-value?
Consider the following two statistical principles: 1) an exact test's $p$-value gives the exact frequency with which the observed random sample appears by chance, i.e., under a true null hypothesis; ...
- 642
1
vote
0
answers
60
views
Can permutation tests only prove non i.i.d-ness?
I just learned about permutation tests through self-study. According to Wikipedia we have :
(1) "The null hypothesis is that all samples come from the same distribution $H_{0}:F=G$."
and :
(...
- 41
2
votes
2
answers
250
views
non-parametric-ANOVA
I have a dataset containing angles. They represent the bending angle that a seedling makes to go toward light. Genotype A is WT and A is the one we are testing. We removed a PKS gene, wich is ...
0
votes
0
answers
69
views
Bootstrapping vs permutations for correlation hypothesis testing
I would like to test the null hypothesis that there is no difference between the outcomes of the Spearmann correlation test run on pairs of data (x,y) from the original dataset I have , vs one where ...
- 79
2
votes
0
answers
58
views
p-value of pearson correlation of correlated samples
I'm not completely sure if this question has been asked before, but I couldn't find one that suit my problem so far.
So basically I am doing single cell analysis, and I have roughly > 100000 cells ...
3
votes
1
answer
55
views
Null hypothesis in permutation inference for the general linear model
I'm currently studying a paper by Winkler et. al on permutation inference for the general linear model and am confused by their reasoning and notation.
They first introduce a formulation for the ...
- 55
1
vote
0
answers
35
views
Which statistical test is suitable to compare order statistics of two independent samples?
Say I want to compare two order statistics (say the 2nd largest value or min value) of two samples. Let's not make any distributional assumptions except that the variance is finite?
Is something like ...
1
vote
0
answers
69
views
How is this a permutation test?
I am trying to read a research paper (link) that involves a permutation test. My (limited) understanding of a permutation test is that you get two samples of data, compute a statistic based on both ...
- 31
3
votes
1
answer
201
views
Tukey's HSD after permutation tests. Theoretically possible?
I'm interested in making more use of tidymodels' infer package, which lets one perform a variety of statistical tests by a simple algorithm. Here's an example pipeline for an F-test, where we aim to <...
1
vote
0
answers
111
views
Permutation testing with LME models
I need to compare two groups of participants to each other in a model that contains fixed and random effects in R. It was suggested to me that a non-parametric test might be ideal for these ...
- 121
2
votes
2
answers
171
views
Multiple regression on unpaired data (A~B+C)
I am not sure how to test one of my hypothesis from my data.
Basically, I have 3 behavioral measures A, B and C, and I want to test that A is a composite of B and C: A = B+C
However, for practical ...
- 21
0
votes
0
answers
28
views
Does bootstrapping along time tell us anything about spatial fields?
A common problem in my field (climate science) is that we want to assess the statistical significance of a 2D spatial field. This field is typically obtained by taking the temporal mean of a subset of ...
- 35
2
votes
0
answers
47
views
Reporting effect size for permutational ANOVA
I ran a permutational one-way ANOVA with blocking. I would like to report the permuted p-values along with an effect size (difference in means). However, the permutational ANOVA does not allow one to ...
- 21
0
votes
0
answers
69
views
Difference between permutations p-value and Z-score p-value
I was asked to find a p-value for difference in means of a given data using 1,000,000 permutations and p-value using z-score of 1,000 permutations. In the beginning I received different results and ...
- 101
2
votes
1
answer
45
views
Difference between two paired non-normal groups
My Data: Respondents were asked to evaluate the quality of two products on a scale of 0-10. There were 12 criteria that constituted the grading scheme, and I would like to analyze the scores overall (...
- 21
0
votes
1
answer
152
views
How to implement multiple comparison correction after Wilcoxon rank-sum test with relativelty small sample sizes and many comparisons
I am comparing 2 groups, performing a total of 180 comparisons. When doing these 180 pair-wise comparisons using the Wilcoxon rank-sum test more than 40 show a significant (p<0.05) difference ...
- 23
1
vote
0
answers
22
views
How to evaluate whether a treatment effect is significant given variance in measurement
I have two samples I am using to evaluate the effect of a treatment. Group $T$ contains 60 treated objects and their scores and group $C$ contains 60 control objects and their scores. The two groups ...
- 115
5
votes
2
answers
294
views
What is the null hypothesis for a randomisation (permutation) test, how should the p-value be interpreted and what conclusions can we draw?
Terminology: The terms permutation and randomisation tests often seem to be used interchangably, but in some cases they have distinct meanings - with the meanings not always consistent. In this post ...
1
vote
0
answers
52
views
Proper way to test threshold-based grouping of data
I have a population data.
I observe two clusters in the distribution of one continuous variable from this population.
I can easily group my data into two clusters with a threshold on this variable.
I ...
- 11
0
votes
0
answers
46
views
procrustes alternative
Im comparing two multidimensional MDS solutions, the solutions have the same number of dimensions. I don't think I can use the permutation version of procrustes analysis (commonly, PROTEST in R::vegan)...
4
votes
1
answer
873
views
Why am I seeing a low F-statistic, but a very significant p-value in permanova?
So I am having some numbers which I can't understand...basically I am performing a permanova test using the adonis2 function, from the vegan package, and what I see ...
- 207
1
vote
0
answers
23
views
Does it make sense to "regress out on one variable" during permutation testing?
In our lab we have developed a non parametric permutation test which assesses whether a feature $f:G\to \mathbb{R}$ respects a given symmetric binary relation $A\subset G\times G$.
Regard $A$ as being ...
0
votes
1
answer
1k
views
Why is the order of factors (order of samples in dataset) affect PERMANOVA results with adonis2?
I am trying to run PERMANOVA tests on multiomics datasets collected from coral samples using adonis2 (v 2.6-5). I found when the order of samples in my dataset change, the pvalues change, sometimes ...
4
votes
1
answer
289
views
Adjusting p-values in multiple regression using permutation test
I have run a multiple regression model using a set of 10 categorical and numerical explanatory variables. This model results in 10 coefficients and their associated p-values. I would like to report ...
- 43
1
vote
0
answers
75
views
Statistic for significance test comparing transition matrix to null
I have several pitch sequences (mini songs), that look something like this when plotted as a pitch profile (piano roll):
I've coded each sequence in terms of its constituent interval sizes, and ...
- 2,332
1
vote
0
answers
76
views
Permutation Test for mean ratio
I am given three vectors of length 11, that represent blood levels of a
hormone measured after placebo, old and new treatment.
...
- 11