Stack Exchange Network

Stack Exchange network consists of 175 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [permutation-test]

Statistical tests based on rearrangements of data that are consistent with the null hypothesis.

2
votes
1answer
14 views

Can we calculate a confidence interval using a permutation testing approach?

Tim Hesterberg's "What Teachers Should Know about the Bootstrap" is excellent. On page 77 he writes: "We cannot use permutation testing to obtain confidence intervals." Is that correct? If so, can ...
0
votes
0answers
49 views

Statistical test for dependent data- comparing groups of pairs

I have a question about statistical tests for dependent data. This will sound a bit strange/extremely naive but pls bear with me- here is my problem formulation: I have 500 individuals, and I'm ...
0
votes
0answers
2 views

Can mantel tests be evaluated on a large stratified matrix or how to combine multiple independent mantel tests for one overall result?

I have 11 pairwise matrices that are all asking the same question. I collected the data at 11 different sites, and what to know if there is an overall relationship (not just at one particular site). ...
1
vote
2answers
58 views

Papers About Permutation Version of Welch's t-test

Permutation tests seem to provide a promising alternative for the unpaired t-test, requiring fewer assumptions. However, the core assumption of the permutation test, exchangeability, implies ...
4
votes
2answers
99 views

Permutation test comparing nested non-linear models with an exchangeable dummy variable

This question is closely related to an earlier question, but I realized my case was actually a lot more specific than the way I formulated it there, in ways that I think merit a separate answer. I ...
0
votes
0answers
19 views

Permutation F test for nested models

How can I do a permutation F test for testing if a larger model which contains a larger model is better? e.g. Model 1 : y ~ x1 + x2 Model 2 : ...
0
votes
1answer
35 views

odd insignificant results using pairwise.perm.manova

I am trying to complete pairwise comparisons of 9 forest communities. Communities were significant at p < 0.001 using adonis. However, when I use the same bray curtis matrix with the pairwise.perm....
0
votes
0answers
36 views

Permutation tests for paired data with several variables

I am trying to answer my own previous question, with a permutation method. I have a reference method $M_{ref}$ that I would like to compare with other methods $M_1$, $M_2$, $M_3$ when taking ...
2
votes
0answers
16 views

Best way to test the difference in means for Tweedie distributions?

What is the best way to test the difference in means for two samples of data that follow a Tweedie distribution? (zero inflated, tweedie power parameter ~1.7) The data represents the number of ...
2
votes
1answer
44 views

What is the point of a permutation F-test when all you need is one F-test for one-way ANOVA?

Say you have three groups, and each group has 5 observations. You can figure out if there is a significant difference between means with a simple one-way ANOVA. I read in my nonparametric book, one ...
0
votes
0answers
12 views

Theoretically is it better to sample with or without replacement to approximate a rank test?

Let's say we want to do a rank test since we do not know the exact distribution. Say we have 3 different sample sizes. Which of the following tests (exact permutation, sample with replacement, sample ...
1
vote
0answers
34 views

How to Calculate a Permutation Test p-value

I've been working through the Hesterberg Bootstrap and Permutation Test Section Here: http://hsta559s12.pbworks.com/w/file/fetch/50747070/Hesterberg.Bootstrappng.chpt18.pdf and I'm stuck on problem 18....
5
votes
0answers
38 views

Advantages of using t-value as test statistic in permutation tests?

I'm working with some permutation tests where my main aim is to evaluate a treatment effect, and I have a question about choice of test statistic. I've seen that some use β for the variable of ...
0
votes
0answers
22 views

Asking opinion about random forests, chi squared and data permutation

I have a set of data of counts of hares in summer and in winter, which is very unbalanced: winter was too cold, and samplings were taken only in some 20 occasions, while during the pleasant summer the ...
0
votes
1answer
24 views

Permutations, shuffling WITHIN but not BETWEEN groups [closed]

In R, I need to make 100 repeat random permutations of group orders, keeping the composition and sizes of groups the same, but the order within the groups needs to be randomly shuffled (100 times) By ...
0
votes
1answer
25 views

hypothesis testing to check for occurrence

I am struggling to understand if I can make such a hypothesis or not and if so, which test is best to validate it. I have a data with a size of 50 (quality of wine) I see that a sensory quality ...
0
votes
0answers
17 views

Permutation testing for simulated compositional data

I'm testing some methods for estimating correlation in compositional data. As part of this process, I'm using the following approach: simulate un-normalized features $Y$ normalize features for each ...
1
vote
1answer
26 views

Using independence_test in R with unequal sample sizes [closed]

This is a homework problem that I have where I'm testing the means of sampled high temperatures in two cities: Des Moines and Chicago. The first part of the question required me to run an unpaired t-...
1
vote
1answer
29 views

What is the proper way to report test statistics used in a permutation analysis?

In my paper, I run some ANOVAs and t-tests, and the test statistics are F and t respectively. I have reason to believe the null distributions are not normal, so I have run permutation tests and ...
1
vote
0answers
58 views

Distribute n balls among k bags [closed]

How many ways can we distribute n balls among k bags if (1) the balls and bags are distinguishable (e.g. numbered). (2) the bags are distinguishable; the balls are not. (3) balls and bags are ...
0
votes
0answers
17 views

which statistical test can be useful without replicate

am wondering if one knows which method is a correct way to go with such a data I have 4 conditions where I obtained the microarray data. I don't have a control and all 4 conditions are treated ones. I ...
1
vote
1answer
67 views

is permutation testing functionally equivalent to training/test?

If permutation testing is applied in machine learning, is permutation testing functionally equivalent to training/test?
1
vote
1answer
177 views

Random forest permutation test: Is permutation of the training set appropriate?

I have a rather high-dimensional data set (p > 1000) with several variables ranking significantly higher than the rest in terms of variable importance (measured by Gini impurity). However, these ...
1
vote
0answers
41 views

FDR for Bayes Factors

I have a dataset, 2 models $M_0$ and $M_1$. Most of my events belong to model $M_0$, so I detect events, belonging to $M_1$, only if the corresponding Bayes factor is bigger than 20. So far so good, ...
0
votes
0answers
10 views

What final test to apply to a permutation test (by compression) algorithm?

I'm following in the steps of §5.1 Permutation Testing, NIST Special Publication 800-90B, Recommendation for the Entropy Sources Used for Random Bit Generation. I can't link to this due to the ...
1
vote
0answers
33 views

Testing for differences between distributions in the real world

I have a data set that consists of measurements of the size and biomass of zooplankton in the Great Lakes. The dataset covers 20 years of sampling in each of the lakes. In essence a net is used to ...
1
vote
0answers
49 views

Number of permutations in Fisher-Pitman test

I would like to compare the means of two independent samples of sizes $n_1 = 2$ and $n_2 = 6$ respectively. I was suggested to use a two-sample Fisher-Pitman permutation test, because it does not ...
0
votes
0answers
14 views

Randomization test for comparing ratio estimators

I want to compare two ratio estimators for fisheries data. They are calculated by summing the catch over years divided by the sum of effort over years. One set of data was collected during an open ...
1
vote
0answers
135 views

Monte Carlo testing: number of required permutations

I want to perform a statistical hypothesis test, however I don't know the exact distribution of my test statistic under $H_0$. Therefore, I need to calculate a Monte Carlo estimate $\hat{p}$ of the ...
0
votes
1answer
133 views

Is there a way to run a multivariate mixed effects model (e.g. RDA) with an unbalanced number of observations per level of the random factor?

I have a community matrix (fish gut microbiome) from a laboratory experiment consisting of 2 treatments, each with 3 independent aquaria, which were sampled 3 times. At each sampling event 6 fish were ...
0
votes
1answer
27 views

randomization/permutation test for mean > 0

I'd like to know if the following procedure is a valid way to test if the mean of a population is greater than zero, based on a random sample of observations: 1) Calculate the sample mean 2) On each ...
1
vote
1answer
60 views

Choose between several confidence interval procedures for randomization test

I am trying to produce confidence intervals for a randomization test of no effect and I've found two different methods in the literature. I have a hard time comparing the two and selecting the best ...
1
vote
0answers
14 views

Analysis of multiple dependent variables all influenced by one indepdenent variable

I am analyzing a psychophysiological experiment. I ran a model on my response times and accuracy data, which gave 7 parameters [dependent variable] per participant ( n = 12) all depending on one ...
2
votes
4answers
3k views

Probabilty of a specific 4 digit number

I'm having an argument with a colleague about the probability of a 4 digit number. The assumption is that the number is randomly and independently created. Assume the probability of generating the ...
0
votes
0answers
98 views

literature about permutation tests in linear mixed models to test fixed effects of categoircal variables under the null hypotheses

I'm currently analyzing a linear mixed model to model the effect of different treatments (two factors, both within and with multiple categories, e.g. treatment1_1, treatmen1_2, treatmen1_3, Treatment ...
4
votes
1answer
74 views

how to deal with correlated/colinear features when using Permutation feature importance?

Permutation feature importance (PFI) is a nice way of getting feature importance in black-box models or models where it is difficult to characterise the relationship between the features and the ...
1
vote
0answers
43 views

Permutation Test under Non-Symmetry (request for literature)

In Firpo (2017), a permutation testing procedure is used for causal inference on a treatment applied to one unit (indexed here by 1). As one would expect, the equation $$p = \frac{1}{J}\sum_{j=1}^JI\{...
2
votes
1answer
127 views

Permutation test for difference in distributions in R

I was reading a paper that says they use an "ANOVA permutation test" to test if there is a difference in the (in-)degree distribution among groups of nodes in a network. I Googled this and found a ...
0
votes
0answers
37 views

How to use permutation distribution in GWAS

I have conducted a genome-wide scan of millions of SNP. I have used a general linear model with a correction to population structure using PCA axes. Looking at the QQ plot of the results I see ...
2
votes
0answers
58 views

Two sample test clustered data, continuous variable with zeros

I have a situation where an experiment is being run in the following manner: A one stage cluster sampling (I think this is accurate description ) is conducted whereby there are multiple organizations ...
1
vote
0answers
125 views

2-tailed permutation tests for obviously non-symmetric data

Intuitively, the numerator in the pvalue calculation for 2-sided permutation test involves counting the number of permuted values more extreme (either larger or smaller) than the true value. If the ...
1
vote
0answers
42 views

Trying to find solution for faster randomization inference code in R [closed]

I'm looking for a faster way to run randomization inference in R. A template of my common code for randomization inference is below. It works fine, but when the dataset is large, the for loop can ...
0
votes
0answers
226 views

further question on “simulate.p.value option in chisq.test in R ”

Thank you very much, Glen_b, for your reply to my question in the link, (simulate.p.value option in chisq.test in R). However, still I am confused in one point, you said that "however the fact that ...
0
votes
1answer
1k views

Can you perform a PERMANOVA analysis on nested data?

I've found no answer in the papers I have read regarding PERMANOVAs or on the web so I'm really hoping someone could help me answer this question. An example would be testing PERMANOVA on ...
0
votes
0answers
22 views

Distributing elements over multiple buckets of a varying size

If one has b buckets that each have a size of b_i and one has to place n elements in the buckets. How can I mathematically described the amount of different permutations the elements can be placed in ...
1
vote
0answers
34 views

Correction for multiple comparisons after permutation

I've trained 17 regression models using a support vector regression machine providing correlations (r) between true and predicted labels and the mean squared error (MSE) for each model. The target ...
1
vote
0answers
121 views

How to do a permutation test in the coin package [closed]

I'm interested in linguistic units occurring in two different conditions: (i) as utterances (or 'speaking turns') and (ii) as constructed dialog (or 'direct speech'). Here's a sample dataframe: ...
2
votes
1answer
247 views

Help me understand the z-value computed in wilcoxsign_test (R, coin package)?

I'm using the coin-package in R to run comparisons on paired data using wilcoxsign_test. I use the exact version of the test, so I assume that the test goes through all permutations of the paired data ...
0
votes
0answers
43 views

How to test to test if one model is better than another on independent dataset?

I have two models that make predictions on the same independent data set. I would like to make a test and find out if one model is statistically significantly better than another. I've read that ...
1
vote
0answers
116 views

resampling from a frequency distribution (histogram) to test for significance

Suppose I have count data on brands and the number of toys they make. ...