Questions tagged [permutation-test]

Statistical tests based on rearrangements of data that are consistent with the null hypothesis.

335 questions
Filter by
Sorted by
Tagged with
27 views

How can I generate a completely balanced combination

Attaching a code that generates unique combinations of 8 letters but, it still lacks a condition wherein all letters should have equal counts per column. For my code, basically, letter a to h should ...
5 views

In VEGAN ADONIS, what does "method='bray' do when I pass in a distance matrix I already have (UNIFRAC)?

Suppose I have a distance matrix created by applying the weighted unifrac measure. For instance, using a phyloseq object: ...
4 views

Example of Permutations of Objects which are Alike [duplicate]

Can you give an example that let me visualize the formula of the number of permutation of n objects in which k1 are alike, k2 are alike, etc. n!/k1!×k2!×...×kn! where k1! + k2! + ... + kn = n Also ...
23 views

bootstrap testing - p value with subtraction and replication (Python)

from std.disclaimer import "Dammit Jim, I'm a programmer, not a statistician." Edited to try to make the terminology more correct and the question more clear. Mea ...
45 views

Bootstrap for CIs and Permutation Resampling for Hypothesis Test?

In various scattered places online and StackExchange I've read that bootstrap resampling is more appropriate for calculating confidence intervals while permutation resampling is more appropriate for ...
62 views

Two-tailed one-sample permutation test

I want to test the significance of the classification accuracy of a cross-validated binary classifier. I performed a permutation test to determine whether the true classification accuracy is higher ...
27 views

Using a permutation test to test for differences in the median between two groups when distributions of the variable of interest are not equal

in the following, I will define (1) the hypothesis I wanted to test, (2) the method I used, and (3) the problems that I feel are present. My dataset consists of measures of one dependent variable for ...
49 views

Is two group comparison test combinig rarefaction and permutation usefull and correct?

I wish to know if the analytical procedure using R environment described below is statistically coherent to enable interpretations and conclusions. I looking for evaluations and opinions about the ...
7 views

Compute effect size from simulated NULL distribution

I computed a empirical p-value (permutation test) where I compare my real value to simulated values (N=100,000). In the end I've a distribution like this (vertical line = my real value). For the p-...
32 views

Is this a valid non-parametric approach for computing false discovery rate (FDR)?

Summarizing the method in the reference below as I understand it: For each of $N$ "gene sets" (where a gene set represents a single null hypothesis significance test), compute a test statistic $T_i$. ...
66 views

How did these researchers determine the confidence interval of the AUROC using resampling but without retraining the model?

In this Nature article backed by Google, the investigators develop then externally validate a deep learning model for predicting lung cancer using CT scans. In their internal validation results, we ...
18 views

Is a group structure required for randomization tests?

In randomization tests (for example in Testing Statistical Hypothesis Theorem 15.2.1 Lehmann and Romano) it is required that we have a group of transformations $\mathbf{G}$ on which $X$ and $gX$ have ...
24 views

Difference between three randomization methods for permutation test

Source: http://www.lock5stat.com/StatKey/randomization_1_quant_1_cat/randomization_1_quant_1_cat.html There are three randomization methods for conducting the permutation test which is a non-...
102 views

Comparison of classification accuracy with permutation tests [closed]

My dataset contains 4 balanced classes (A, B, C and D). By carrying out two independent binary classifications, I want to compare the classification accuracy of (A vs. B) with that of (C vs. D). ...
17 views

How to calculate the significance of slope and y-intercept using a permutation test of pairwise fst x distance

I have two variables, 1. scaled pairwise Fst (measure of genetic differentiation between two sites) 2. river kilometers between the two sites. The values are not independent (all of the sites are ...
20 views

Do randomization tests for difference in means assume independent groups?

I performed a randomization test and found that the means of groups A and B are significantly different. I am trying to prove that group B is "special", that there is something interesting going on ...
30 views

Randomization test vs. t-test vs. others?

I am trying to find out if the means of two groups are different. I have performed a randomization test and found significant results but I would like to confirm it with some other test. I looked into ...
33 views

Permutation test for accuracy in multiclass discrimination

It is well known that to assess the significance (i.e. difference from chance) of the performance of a classification analysis one can use a permutation test. In a two-classes case, the procedure is ...
25 views

Correction for multiple testing of permutation p-values

I simultaneously performed multiple statistical tests on my samples under investigation. For each test i obtained p-value that i had to correct for ...
9 views

Re-randomization for cluster randomized trial, implications for primary analysis

I am hoping the masses might have some insight or a couple papers to point to on the issue I am facing. My research group conducts cluster-randomized trials where we assign treatment by cluster (...
11 views

Is it mathematically sound to use binary independent variables in a permutational ANOVA since it does not assume multivariate normality?

I am trying to determine if it is mathematically sound to use binary independent variables in a permutational ANOVA since it does not assume multivariate normality. I was considering Canonical ...
44 views

How to calculate permutation test p-value when resulting computation of permutations is a matrix of test statistics

I am working with 2 matrices consisting of matrix 1 (xrows, ycols) and matrix 2 (yrows, zcols). My resulting computation is a matrices multiplication which results in a matrix of (xrows, zcols). I ...
18 views

Difference in results of permutation tests through coin::oneway.test and resample::permutationTest2

I'm trying to run a same permutation test for the null hypothesis of no difference between means with the coin and resample R ...
23 views

RDA axes significance

I performed a RDA with RDA function from vegan library. I tested model significance with permutations and I also teste axes significance. Results indicate that the model is significant (p=.005). From ...
29 views

How to calculate a permutation confidence interval?

Mara Tableman, Minh Nguyen, and Michael D. Ernst note The idea behind a permutation confidence interval is well known and fairly straightforward (Ernst 2004). The confidence interval is simply the ...
72 views

Can we calculate a confidence interval using a permutation testing approach?

Tim Hesterberg's "What Teachers Should Know about the Bootstrap" is excellent. On page 77 he writes: "We cannot use permutation testing to obtain confidence intervals." Is that correct? If so, can ...
53 views

Statistical test for dependent data- comparing groups of pairs

I have a question about statistical tests for dependent data. This will sound a bit strange/extremely naive but pls bear with me- here is my problem formulation: I have 500 individuals, and I'm ...
5 views

Can mantel tests be evaluated on a large stratified matrix or how to combine multiple independent mantel tests for one overall result?

I have 11 pairwise matrices that are all asking the same question. I collected the data at 11 different sites, and what to know if there is an overall relationship (not just at one particular site). ...
75 views

Papers About Permutation Version of Welch's t-test

Permutation tests seem to provide a promising alternative for the unpaired t-test, requiring fewer assumptions. However, the core assumption of the permutation test, exchangeability, implies ...
112 views

Permutation test comparing nested non-linear models with an exchangeable dummy variable

This question is closely related to an earlier question, but I realized my case was actually a lot more specific than the way I formulated it there, in ways that I think merit a separate answer. I ...
30 views

Permutation F test for nested models

How can I do a permutation F test for testing if a larger model which contains a larger model is better? e.g. Model 1 : y ~ x1 + x2 Model 2 : ...
82 views

odd insignificant results using pairwise.perm.manova

I am trying to complete pairwise comparisons of 9 forest communities. Communities were significant at p < 0.001 using adonis. However, when I use the same bray curtis matrix with the pairwise.perm....
51 views

Permutation tests for paired data with several variables

I am trying to answer my own previous question, with a permutation method. I have a reference method $M_{ref}$ that I would like to compare with other methods $M_1$, $M_2$, $M_3$ when taking ...
32 views

Best way to test the difference in means for Tweedie distributions?

What is the best way to test the difference in means for two samples of data that follow a Tweedie distribution? (zero inflated, tweedie power parameter ~1.7) The data represents the number of ...
70 views

What is the point of a permutation F-test when all you need is one F-test for one-way ANOVA?

Say you have three groups, and each group has 5 observations. You can figure out if there is a significant difference between means with a simple one-way ANOVA. I read in my nonparametric book, one ...
28 views

Theoretically is it better to sample with or without replacement to approximate a rank test?

Let's say we want to do a rank test since we do not know the exact distribution. Say we have 3 different sample sizes. Which of the following tests (exact permutation, sample with replacement, sample ...
44 views

How to Calculate a Permutation Test p-value

I've been working through the Hesterberg Bootstrap and Permutation Test Section Here: http://hsta559s12.pbworks.com/w/file/fetch/50747070/Hesterberg.Bootstrappng.chpt18.pdf and I'm stuck on problem 18....
50 views

Advantages of using t-value as test statistic in permutation tests?

I'm working with some permutation tests where my main aim is to evaluate a treatment effect, and I have a question about choice of test statistic. I've seen that some use β for the variable of ...
32 views

Asking opinion about random forests, chi squared and data permutation

I have a set of data of counts of hares in summer and in winter, which is very unbalanced: winter was too cold, and samplings were taken only in some 20 occasions, while during the pleasant summer the ...
30 views

Permutations, shuffling WITHIN but not BETWEEN groups [closed]

In R, I need to make 100 repeat random permutations of group orders, keeping the composition and sizes of groups the same, but the order within the groups needs to be randomly shuffled (100 times) By ...
26 views

hypothesis testing to check for occurrence

I am struggling to understand if I can make such a hypothesis or not and if so, which test is best to validate it. I have a data with a size of 50 (quality of wine) I see that a sensory quality ...
26 views

Permutation testing for simulated compositional data

I'm testing some methods for estimating correlation in compositional data. As part of this process, I'm using the following approach: simulate un-normalized features $Y$ normalize features for each ...
38 views

Using independence_test in R with unequal sample sizes [closed]

This is a homework problem that I have where I'm testing the means of sampled high temperatures in two cities: Des Moines and Chicago. The first part of the question required me to run an unpaired t-...
106 views

What is the proper way to report test statistics used in a permutation analysis?

In my paper, I run some ANOVAs and t-tests, and the test statistics are F and t respectively. I have reason to believe the null distributions are not normal, so I have run permutation tests and ...
17 views

which statistical test can be useful without replicate

am wondering if one knows which method is a correct way to go with such a data I have 4 conditions where I obtained the microarray data. I don't have a control and all 4 conditions are treated ones. I ...
162 views

is permutation testing functionally equivalent to training/test?

If permutation testing is applied in machine learning, is permutation testing functionally equivalent to training/test?
274 views

Random forest permutation test: Is permutation of the training set appropriate?

I have a rather high-dimensional data set (p > 1000) with several variables ranking significantly higher than the rest in terms of variable importance (measured by Gini impurity). However, these ...
68 views

FDR for Bayes Factors

I have a dataset, 2 models $M_0$ and $M_1$. Most of my events belong to model $M_0$, so I detect events, belonging to $M_1$, only if the corresponding Bayes factor is bigger than 20. So far so good, ...