Skip to main content

Questions tagged [permutation-test]

Statistical tests based on rearrangements of data that are consistent with the null hypothesis.

Filter by
Sorted by
Tagged with
2 votes
1 answer
49 views

mean of medians consistently differs from median

Disclaimer: my first question here. Background of the problem: I want to compare two distributions (n1 ~ 500 and n2 ~ 700), not normal and with different variances, but roughly unimodal. I decided to ...
Paul S's user avatar
  • 21
1 vote
1 answer
26 views

Estimating Confidence in Feature Rankings from Multiple Experiments with Non-Normal Data

Hello dear Cross Validated Community, I am a new doctoral student in bioinformatics, and I am working on a project involving multiple experiments, each generating a single numerical result for each of ...
Thomas Rauter's user avatar
0 votes
0 answers
7 views

Using the jackknife method in permutation tests

We aim to employ a permutation test to evaluate the significance of the relationship between two datasets using Sparse Canonical Correlation Analysis (SCCA) for a specific level of sparsity. To ...
R.N.'s user avatar
  • 1
1 vote
0 answers
38 views

Plotting the mean difference of two different sample sizes, with the effect size in R

I have a simple data frame of approximately 1750 participants. But have taken the first 11 participants to form the example. From the all_groups variable, I have 4 ...
user avatar
0 votes
0 answers
11 views

Proportion testing from pooled single cell/nucleus samples

I have a snRNAseq experiment with a 2x2 layout (wt vs transgenic, drug vs vehicle; Wt-V, Wt-D, Tg-V, Tg-D). I have 6 mice per group, but for cost reasons we pooled pairs of samples in each group, for ...
dragon951's user avatar
  • 153
1 vote
0 answers
25 views

Can p-values from permutation tests be interpreted as a measure of effect size between group?

I'm conducting a research project involving gene expression analysis. We've developed a method to calculate a score from a given set of genes, A. To evaluate the ...
zhang's user avatar
  • 113
1 vote
0 answers
16 views

Is a permutation test valid in comparing a fraud block policy false positive rate across two different months?

I'm looking to conduct a counterfactual experiment in an e-commerce fraud context. When a fraud policy blocks a transaction you necessarily will never know if it was truly fraudulent. I'm trying to ...
phx's user avatar
  • 11
0 votes
0 answers
35 views

Algorithm for conducting Westfall & Young permutation test under Multiple Hypothesis Testing given a list of raw(unadjusted) p values

I am currently trying to set up a python program to implement Westfall & Young (1993)'s minP permutation procedure using Python. The computationally efficient algorithm I was able to find by ...
Suzorro's user avatar
1 vote
0 answers
28 views

Repeated measures microbiome data with baseline measurement: comparison of two treatments

I am interested in any advice I can get on different ways I might analyze multivariate ecological community (fecal sample microbiome) data that come from a "before after control impact" ...
qdread's user avatar
  • 295
0 votes
0 answers
14 views

Necessary assumptions for a permutation test of conditional independence

Consider a case in which you want to know if two variables (X, Y) are independent conditional given a set (C) of other variables. A recent paper (Shah, R. D., and J. Peters. 2020. The hardness of ...
Bill Shipley's user avatar
1 vote
0 answers
51 views

What is the connection between permutation tests and rank tests?

I was trying to understand the null hypothesis of the Wilcoxon rank sum test and thought that the null distribution of the rank sum could be obtained by a permutation test/Monte Carlo test. If this ...
Ggjj11's user avatar
  • 1,373
0 votes
1 answer
41 views

Why is the order of samples in my metadata changing my PERMANOVA results with adonis?

I've trying to preform a PERMANOVA using adonis2 on a dissimilarity matrix. My code looks like this. ...
CassandraRegine's user avatar
0 votes
0 answers
14 views

Sanity checking custom cluster extent permutation test of time series data correctly

I am currently developing a cluster extent permutation test on time series data. For that I wanted to sanity test to see whether there are any biases with my test when it is ran on null (i.e., no ...
Mah1510's user avatar
  • 11
0 votes
0 answers
38 views

correct formula for permutation test [duplicate]

in order I need do: measuring the targeted variable on the original network and save it Shuffling my original data (correlation matrix) - randomly (100x) create the graph measuring the targeted ...
Ana Paula Castro's user avatar
1 vote
1 answer
88 views

Test for statistically significant difference: Wilcoxon vs Median Permutation

I have a dataset that shows measured 'perceived quality'(dependent variable), by two different sizes in the context of 4 different intensity levels (200 values for perceived quality in each category, ...
Paris 's user avatar
  • 13
6 votes
1 answer
198 views

Permutation test for exponential null hypothesis: really bad?

Having found nice formulas for testing the null hypothesis under exponentially-distributed samples, I wanted to see how well permutation tests could do the job. And the answer, assuming no mistakes, ...
feetwet's user avatar
  • 1,148
1 vote
1 answer
66 views

When and why is a likelihood ratio preferable to a difference as a test statistic?

We want to know whether two sample sets {x} and {y} were drawn from the same distribution. The null hypothesis $H_0$ is that they are. As statisticians we test the hypothesis by calculating the p-...
feetwet's user avatar
  • 1,148
1 vote
0 answers
30 views

Is a permutation test with max statistics appropriate to account for multiple comparisons when searching for correlations between two questionnaires?

I'm trying to understand if the max statistic method for permutation test is appropriate to use and how i should apply it in my situation: I have a sample of 38 individuals who performed a task and we ...
Francesco Favro's user avatar
6 votes
4 answers
571 views

What is the information in an exact p-value?

Consider the following two statistical principles: 1) an exact test's $p$-value gives the exact frequency with which the observed random sample appears by chance, i.e., under a true null hypothesis; ...
virtuolie's user avatar
  • 580
1 vote
0 answers
57 views

Can permutation tests only prove non i.i.d-ness?

I just learned about permutation tests through self-study. According to Wikipedia we have : (1) "The null hypothesis is that all samples come from the same distribution $H_{0}:F=G$." and : (...
DeadKarlMarx's user avatar
2 votes
2 answers
248 views

non-parametric-ANOVA

I have a dataset containing angles. They represent the bending angle that a seedling makes to go toward light. Genotype A is WT and A is the one we are testing. We removed a PKS gene, wich is ...
Marius Audenis's user avatar
0 votes
0 answers
53 views

Bootstrapping vs permutations for correlation hypothesis testing

I would like to test the null hypothesis that there is no difference between the outcomes of the Spearmann correlation test run on pairs of data (x,y) from the original dataset I have , vs one where ...
aim6789's user avatar
  • 77
2 votes
0 answers
55 views

p-value of pearson correlation of correlated samples

I'm not completely sure if this question has been asked before, but I couldn't find one that suit my problem so far. So basically I am doing single cell analysis, and I have roughly > 100000 cells ...
Andreas Adinatha's user avatar
3 votes
1 answer
54 views

Null hypothesis in permutation inference for the general linear model

I'm currently studying a paper by Winkler et. al on permutation inference for the general linear model and am confused by their reasoning and notation. They first introduce a formulation for the ...
Adrian Mak's user avatar
1 vote
0 answers
33 views

Which statistical test is suitable to compare order statistics of two independent samples?

Say I want to compare two order statistics (say the 2nd largest value or min value) of two samples. Let's not make any distributional assumptions except that the variance is finite? Is something like ...
Estimate the estimators's user avatar
1 vote
0 answers
63 views

How is this a permutation test?

I am trying to read a research paper (link) that involves a permutation test. My (limited) understanding of a permutation test is that you get two samples of data, compute a statistic based on both ...
caitlin's user avatar
  • 31
0 votes
0 answers
24 views

How to analyze within-subjects repeated measures design with more data for some participants?

I ran a within-subjects study where the study was broken into four time periods and participants were allowed to participate once in each time period. Each time they participated they were exposed to ...
tree's user avatar
  • 113
3 votes
1 answer
146 views

Tukey's HSD after permutation tests. Theoretically possible?

I'm interested in making more use of tidymodels' infer package, which lets one perform a variety of statistical tests by a simple algorithm. Here's an example pipeline for an F-test, where we aim to <...
h088bmIuXaskpzJEe3ld's user avatar
1 vote
0 answers
96 views

Permutation testing with LME models

I need to compare two groups of participants to each other in a model that contains fixed and random effects in R. It was suggested to me that a non-parametric test might be ideal for these ...
Shannon Cahalan's user avatar
2 votes
2 answers
156 views

Multiple regression on unpaired data (A~B+C)

I am not sure how to test one of my hypothesis from my data. Basically, I have 3 behavioral measures A, B and C, and I want to test that A is a composite of B and C: A = B+C However, for practical ...
FloLe's user avatar
  • 21
0 votes
0 answers
26 views

Does bootstrapping along time tell us anything about spatial fields?

A common problem in my field (climate science) is that we want to assess the statistical significance of a 2D spatial field. This field is typically obtained by taking the temporal mean of a subset of ...
boro141's user avatar
  • 35
1 vote
0 answers
35 views

Reporting effect size for permutational ANOVA

I ran a permutational one-way ANOVA with blocking. I would like to report the permuted p-values along with an effect size (difference in means). However, the permutational ANOVA does not allow one to ...
aibiodiv's user avatar
0 votes
0 answers
56 views

Difference between permutations p-value and Z-score p-value

I was asked to find a p-value for difference in means of a given data using 1,000,000 permutations and p-value using z-score of 1,000 permutations. In the beginning I received different results and ...
Zeev's user avatar
  • 101
2 votes
1 answer
41 views

Difference between two paired non-normal groups

My Data: Respondents were asked to evaluate the quality of two products on a scale of 0-10. There were 12 criteria that constituted the grading scheme, and I would like to analyze the scores overall (...
Derek 's user avatar
  • 21
0 votes
1 answer
98 views

How to implement multiple comparison correction after Wilcoxon rank-sum test with relativelty small sample sizes and many comparisons

I am comparing 2 groups, performing a total of 180 comparisons. When doing these 180 pair-wise comparisons using the Wilcoxon rank-sum test more than 40 show a significant (p<0.05) difference ...
ArthurDent's user avatar
1 vote
0 answers
22 views

How to evaluate whether a treatment effect is significant given variance in measurement

I have two samples I am using to evaluate the effect of a treatment. Group $T$ contains 60 treated objects and their scores and group $C$ contains 60 control objects and their scores. The two groups ...
naveace's user avatar
  • 105
5 votes
2 answers
264 views

What is the null hypothesis for a randomisation (permutation) test, how should the p-value be interpreted and what conclusions can we draw?

Terminology: The terms permutation and randomisation tests often seem to be used interchangably, but in some cases they have distinct meanings - with the meanings not always consistent. In this post ...
8e9yQBKVlIDwoIVegfkJ's user avatar
1 vote
0 answers
47 views

Proper way to test threshold-based grouping of data

I have a population data. I observe two clusters in the distribution of one continuous variable from this population. I can easily group my data into two clusters with a threshold on this variable. I ...
Gsecer's user avatar
  • 11
0 votes
0 answers
41 views

procrustes alternative

Im comparing two multidimensional MDS solutions, the solutions have the same number of dimensions. I don't think I can use the permutation version of procrustes analysis (commonly, PROTEST in R::vegan)...
EAAndersson's user avatar
4 votes
1 answer
800 views

Why am I seeing a low F-statistic, but a very significant p-value in permanova?

So I am having some numbers which I can't understand...basically I am performing a permanova test using the adonis2 function, from the vegan package, and what I see ...
gabt's user avatar
  • 207
1 vote
0 answers
23 views

Does it make sense to "regress out on one variable" during permutation testing?

In our lab we have developed a non parametric permutation test which assesses whether a feature $f:G\to \mathbb{R}$ respects a given symmetric binary relation $A\subset G\times G$. Regard $A$ as being ...
Patrick Nicodemus's user avatar
0 votes
1 answer
968 views

Why is the order of factors (order of samples in dataset) affect PERMANOVA results with adonis2?

I am trying to run PERMANOVA tests on multiomics datasets collected from coral samples using adonis2 (v 2.6-5). I found when the order of samples in my dataset change, the pvalues change, sometimes ...
amandarwills's user avatar
4 votes
1 answer
234 views

Adjusting p-values in multiple regression using permutation test

I have run a multiple regression model using a set of 10 categorical and numerical explanatory variables. This model results in 10 coefficients and their associated p-values. I would like to report ...
lmb8's user avatar
  • 43
1 vote
0 answers
70 views

Statistic for significance test comparing transition matrix to null

I have several pitch sequences (mini songs), that look something like this when plotted as a pitch profile (piano roll): I've coded each sequence in terms of its constituent interval sizes, and ...
z8080's user avatar
  • 2,322
1 vote
0 answers
73 views

Permutation Test for mean ratio

I am given three vectors of length 11, that represent blood levels of a hormone measured after placebo, old and new treatment. ...
Joshua's user avatar
  • 11
2 votes
1 answer
138 views

How do you use numPerms to get the total number of unique permutations for a PERMANOVA?

I am trying to calculate the number of unique permutations for a PERMANOVA using the R adonis2 function but I am getting stuck on defining my permutation scheme. As ...
planktonQ's user avatar
2 votes
1 answer
217 views

How do I find the number of unique permutations for a PERMANOVA with an unbalanced design?

From the PRIMER manual (p. 28 about Monte Carlo permutations; http://updates.primer-e.com/primer7/manuals/PERMANOVA+_manual.pdf), the number of unique permutations for a PERMANOVA with $a$ groups and $...
planktonQ's user avatar
2 votes
1 answer
569 views

What kind of null hypothesis can a permutation test consider?

From what I have seen, permutation tests are always used to compare the means (or median) of two different groups in the experiment. Even Wikipedia says A permutation test involves two or more ...
GaryTheBaddy's user avatar
1 vote
0 answers
112 views

Can we apply a permutation test to compare two percentage distributions?

Any help here will be greatly appreciated! Suppose we have two groups, one composed of ET's from Mars and another from ET's from Jupiter (assuming there are ETs there). Then, these groups of ETs meet ...
Jean Tardelli's user avatar
4 votes
1 answer
732 views

Can I use a permutation test on timeseries data?

What I am trying to do: I am currently doing analysis on neuronal calcium imaging data. In particular, I have two things: A time series that represents the amount of calcium within a neuron A boolean ...
Luca's user avatar
  • 43

1
2 3 4 5
10