Questions tagged [permutation-test]

Statistical tests based on rearrangements of data that are consistent with the null hypothesis.

Filter by
Sorted by
Tagged with
0
votes
1answer
27 views

How can I generate a completely balanced combination

Attaching a code that generates unique combinations of 8 letters but, it still lacks a condition wherein all letters should have equal counts per column. For my code, basically, letter a to h should ...
0
votes
0answers
5 views

In VEGAN ADONIS, what does "method='bray' do when I pass in a distance matrix I already have (UNIFRAC)?

Suppose I have a distance matrix created by applying the weighted unifrac measure. For instance, using a phyloseq object: ...
0
votes
0answers
4 views

Example of Permutations of Objects which are Alike [duplicate]

Can you give an example that let me visualize the formula of the number of permutation of n objects in which k1 are alike, k2 are alike, etc. n!/k1!×k2!×...×kn! where k1! + k2! + ... + kn = n Also ...
0
votes
0answers
23 views

bootstrap testing - p value with subtraction and replication (Python)

from std.disclaimer import "Dammit Jim, I'm a programmer, not a statistician." Edited to try to make the terminology more correct and the question more clear. Mea ...
1
vote
2answers
45 views

Bootstrap for CIs and Permutation Resampling for Hypothesis Test?

In various scattered places online and StackExchange I've read that bootstrap resampling is more appropriate for calculating confidence intervals while permutation resampling is more appropriate for ...
2
votes
1answer
62 views

Two-tailed one-sample permutation test

I want to test the significance of the classification accuracy of a cross-validated binary classifier. I performed a permutation test to determine whether the true classification accuracy is higher ...
1
vote
0answers
27 views

Using a permutation test to test for differences in the median between two groups when distributions of the variable of interest are not equal

in the following, I will define (1) the hypothesis I wanted to test, (2) the method I used, and (3) the problems that I feel are present. My dataset consists of measures of one dependent variable for ...
0
votes
0answers
49 views

Is two group comparison test combinig rarefaction and permutation usefull and correct?

I wish to know if the analytical procedure using R environment described below is statistically coherent to enable interpretations and conclusions. I looking for evaluations and opinions about the ...
0
votes
0answers
7 views

Compute effect size from simulated NULL distribution

I computed a empirical p-value (permutation test) where I compare my real value to simulated values (N=100,000). In the end I've a distribution like this (vertical line = my real value). For the p-...
2
votes
0answers
32 views

Is this a valid non-parametric approach for computing false discovery rate (FDR)?

Summarizing the method in the reference below as I understand it: For each of $N$ "gene sets" (where a gene set represents a single null hypothesis significance test), compute a test statistic $T_i$. ...
3
votes
1answer
66 views

How did these researchers determine the confidence interval of the AUROC using resampling but without retraining the model?

In this Nature article backed by Google, the investigators develop then externally validate a deep learning model for predicting lung cancer using CT scans. In their internal validation results, we ...
1
vote
0answers
18 views

Is a group structure required for randomization tests?

In randomization tests (for example in Testing Statistical Hypothesis Theorem 15.2.1 Lehmann and Romano) it is required that we have a group of transformations $\mathbf{G}$ on which $X$ and $gX$ have ...
1
vote
0answers
24 views

Difference between three randomization methods for permutation test

Source: http://www.lock5stat.com/StatKey/randomization_1_quant_1_cat/randomization_1_quant_1_cat.html There are three randomization methods for conducting the permutation test which is a non-...
1
vote
0answers
102 views

Comparison of classification accuracy with permutation tests [closed]

My dataset contains 4 balanced classes (A, B, C and D). By carrying out two independent binary classifications, I want to compare the classification accuracy of (A vs. B) with that of (C vs. D). ...
0
votes
0answers
17 views

How to calculate the significance of slope and y-intercept using a permutation test of pairwise fst x distance

I have two variables, 1. scaled pairwise Fst (measure of genetic differentiation between two sites) 2. river kilometers between the two sites. The values are not independent (all of the sites are ...
0
votes
1answer
20 views

Do randomization tests for difference in means assume independent groups?

I performed a randomization test and found that the means of groups A and B are significantly different. I am trying to prove that group B is "special", that there is something interesting going on ...
0
votes
1answer
30 views

Randomization test vs. t-test vs. others?

I am trying to find out if the means of two groups are different. I have performed a randomization test and found significant results but I would like to confirm it with some other test. I looked into ...
0
votes
0answers
33 views

Permutation test for accuracy in multiclass discrimination

It is well known that to assess the significance (i.e. difference from chance) of the performance of a classification analysis one can use a permutation test. In a two-classes case, the procedure is ...
0
votes
0answers
25 views

Correction for multiple testing of permutation p-values

I simultaneously performed multiple statistical tests on my samples under investigation. For each test i obtained p-value that i had to correct for ...
0
votes
0answers
9 views

Re-randomization for cluster randomized trial, implications for primary analysis

I am hoping the masses might have some insight or a couple papers to point to on the issue I am facing. My research group conducts cluster-randomized trials where we assign treatment by cluster (...
0
votes
0answers
11 views

Is it mathematically sound to use binary independent variables in a permutational ANOVA since it does not assume multivariate normality?

I am trying to determine if it is mathematically sound to use binary independent variables in a permutational ANOVA since it does not assume multivariate normality. I was considering Canonical ...
1
vote
0answers
44 views

How to calculate permutation test p-value when resulting computation of permutations is a matrix of test statistics

I am working with 2 matrices consisting of matrix 1 (xrows, ycols) and matrix 2 (yrows, zcols). My resulting computation is a matrices multiplication which results in a matrix of (xrows, zcols). I ...
0
votes
0answers
18 views

Difference in results of permutation tests through coin::oneway.test and resample::permutationTest2

I'm trying to run a same permutation test for the null hypothesis of no difference between means with the coin and resample R ...
0
votes
0answers
23 views

RDA axes significance

I performed a RDA with RDA function from vegan library. I tested model significance with permutations and I also teste axes significance. Results indicate that the model is significant (p=.005). From ...
1
vote
0answers
29 views

How to calculate a permutation confidence interval?

Mara Tableman, Minh Nguyen, and Michael D. Ernst note The idea behind a permutation confidence interval is well known and fairly straightforward (Ernst 2004). The confidence interval is simply the ...
2
votes
3answers
72 views

Can we calculate a confidence interval using a permutation testing approach?

Tim Hesterberg's "What Teachers Should Know about the Bootstrap" is excellent. On page 77 he writes: "We cannot use permutation testing to obtain confidence intervals." Is that correct? If so, can ...
0
votes
0answers
53 views

Statistical test for dependent data- comparing groups of pairs

I have a question about statistical tests for dependent data. This will sound a bit strange/extremely naive but pls bear with me- here is my problem formulation: I have 500 individuals, and I'm ...
0
votes
0answers
5 views

Can mantel tests be evaluated on a large stratified matrix or how to combine multiple independent mantel tests for one overall result?

I have 11 pairwise matrices that are all asking the same question. I collected the data at 11 different sites, and what to know if there is an overall relationship (not just at one particular site). ...
1
vote
2answers
75 views

Papers About Permutation Version of Welch's t-test

Permutation tests seem to provide a promising alternative for the unpaired t-test, requiring fewer assumptions. However, the core assumption of the permutation test, exchangeability, implies ...
3
votes
2answers
112 views

Permutation test comparing nested non-linear models with an exchangeable dummy variable

This question is closely related to an earlier question, but I realized my case was actually a lot more specific than the way I formulated it there, in ways that I think merit a separate answer. I ...
0
votes
0answers
30 views

Permutation F test for nested models

How can I do a permutation F test for testing if a larger model which contains a larger model is better? e.g. Model 1 : y ~ x1 + x2 Model 2 : ...
0
votes
1answer
82 views

odd insignificant results using pairwise.perm.manova

I am trying to complete pairwise comparisons of 9 forest communities. Communities were significant at p < 0.001 using adonis. However, when I use the same bray curtis matrix with the pairwise.perm....
0
votes
0answers
51 views

Permutation tests for paired data with several variables

I am trying to answer my own previous question, with a permutation method. I have a reference method $M_{ref}$ that I would like to compare with other methods $M_1$, $M_2$, $M_3$ when taking ...
2
votes
0answers
32 views

Best way to test the difference in means for Tweedie distributions?

What is the best way to test the difference in means for two samples of data that follow a Tweedie distribution? (zero inflated, tweedie power parameter ~1.7) The data represents the number of ...
2
votes
1answer
70 views

What is the point of a permutation F-test when all you need is one F-test for one-way ANOVA?

Say you have three groups, and each group has 5 observations. You can figure out if there is a significant difference between means with a simple one-way ANOVA. I read in my nonparametric book, one ...
0
votes
0answers
28 views

Theoretically is it better to sample with or without replacement to approximate a rank test?

Let's say we want to do a rank test since we do not know the exact distribution. Say we have 3 different sample sizes. Which of the following tests (exact permutation, sample with replacement, sample ...
1
vote
0answers
44 views

How to Calculate a Permutation Test p-value

I've been working through the Hesterberg Bootstrap and Permutation Test Section Here: http://hsta559s12.pbworks.com/w/file/fetch/50747070/Hesterberg.Bootstrappng.chpt18.pdf and I'm stuck on problem 18....
5
votes
0answers
50 views

Advantages of using t-value as test statistic in permutation tests?

I'm working with some permutation tests where my main aim is to evaluate a treatment effect, and I have a question about choice of test statistic. I've seen that some use β for the variable of ...
0
votes
0answers
32 views

Asking opinion about random forests, chi squared and data permutation

I have a set of data of counts of hares in summer and in winter, which is very unbalanced: winter was too cold, and samplings were taken only in some 20 occasions, while during the pleasant summer the ...
0
votes
1answer
30 views

Permutations, shuffling WITHIN but not BETWEEN groups [closed]

In R, I need to make 100 repeat random permutations of group orders, keeping the composition and sizes of groups the same, but the order within the groups needs to be randomly shuffled (100 times) By ...
0
votes
1answer
26 views

hypothesis testing to check for occurrence

I am struggling to understand if I can make such a hypothesis or not and if so, which test is best to validate it. I have a data with a size of 50 (quality of wine) I see that a sensory quality ...
0
votes
0answers
26 views

Permutation testing for simulated compositional data

I'm testing some methods for estimating correlation in compositional data. As part of this process, I'm using the following approach: simulate un-normalized features $Y$ normalize features for each ...
1
vote
1answer
38 views

Using independence_test in R with unequal sample sizes [closed]

This is a homework problem that I have where I'm testing the means of sampled high temperatures in two cities: Des Moines and Chicago. The first part of the question required me to run an unpaired t-...
1
vote
1answer
106 views

What is the proper way to report test statistics used in a permutation analysis?

In my paper, I run some ANOVAs and t-tests, and the test statistics are F and t respectively. I have reason to believe the null distributions are not normal, so I have run permutation tests and ...
0
votes
0answers
17 views

which statistical test can be useful without replicate

am wondering if one knows which method is a correct way to go with such a data I have 4 conditions where I obtained the microarray data. I don't have a control and all 4 conditions are treated ones. I ...
1
vote
1answer
162 views

is permutation testing functionally equivalent to training/test?

If permutation testing is applied in machine learning, is permutation testing functionally equivalent to training/test?
1
vote
1answer
274 views

Random forest permutation test: Is permutation of the training set appropriate?

I have a rather high-dimensional data set (p > 1000) with several variables ranking significantly higher than the rest in terms of variable importance (measured by Gini impurity). However, these ...
1
vote
0answers
68 views

FDR for Bayes Factors

I have a dataset, 2 models $M_0$ and $M_1$. Most of my events belong to model $M_0$, so I detect events, belonging to $M_1$, only if the corresponding Bayes factor is bigger than 20. So far so good, ...
1
vote
0answers
15 views

What final test to apply to a permutation test (by compression) algorithm?

I'm following in the steps of §5.1 Permutation Testing, NIST Special Publication 800-90B, Recommendation for the Entropy Sources Used for Random Bit Generation. I'm developing my test for ...
1
vote
0answers
41 views

Testing for differences between distributions in the real world

I have a data set that consists of measurements of the size and biomass of zooplankton in the Great Lakes. The dataset covers 20 years of sampling in each of the lakes. In essence a net is used to ...