Questions tagged [permutation-test]

Statistical tests based on rearrangements of data that are consistent with the null hypothesis.

Filter by
Sorted by
Tagged with
0 votes
0 answers
7 views

How to tally when the permuted test statistic = original test statistic during permutation testing?

I'm doing some permutation testing, and in my case the test statistic is an integer value. So how do I tally the situation where Tpermuted = Toriginal? In that case Tpermuted is neither greater nor ...
0 votes
0 answers
4 views

I need a simple but indepth explanation of what a perMANOVA is and how to use it

I've been told by the supervisors of my honors project that I'll be using a perMANOVA to analyze the differences in invertebrate abundance and diversity across 4 different sites, some of which have ...
1 vote
1 answer
44 views

permutation tests of group differences when data contain repeated measures

I have data from several mice with id id. Each mouse either belongs to either group C or group ...
  • 1,037
3 votes
1 answer
53 views

Permutations with repeated measures

Let's say I have data from several mice with id id. Each mouse wandered around two mazes (A and ...
  • 1,037
0 votes
0 answers
20 views

Is it possible to pool results from PERMANOVA tests on multiply imputed datasets?

I wondered whether it is possible to pool results from multiple PERMANOVA tests performed on imputed data. In case of ANOVA, methods of how to pool results have been proposed and discussed, e.g. van ...
1 vote
0 answers
10 views

permutation test for mixed effects 3-stage nested design

I have a dataset that uses a 3-stage nested design, and I want to perform an ANOVA. I used lme() from the nlme package in R to fit a linear model, but I found that after doing a shapiro-Wilks test, ...
  • 11
0 votes
0 answers
17 views

Variable Importance Methods for Multiple Variables/Features simultaneously

I have fitted a classifier on a number of input variables/features that can be reasonably grouped. I am interested in finding out the importance of sets/groups of variables (as opposed to individual ...
1 vote
1 answer
28 views

Independence-violation issue in pairwise measurements: Which test?

Here is another problem in which independence might be an issue (however, the question might be very basic). Suppose, I have a rowing team and I have two boats (B1 and B2, for brevity). I have strong ...
  • 11
0 votes
0 answers
24 views

Permutation test to overcome autocorrelation issues?

let's say, I want to track a rowing team's (collective) performance on some task and show that it increases with practice. I have a metric, i.e. the time to measure their performance. Now I would like ...
  • 11
0 votes
0 answers
23 views

how to determine significance with permutation t-distribution values

suppose I have the following array of t-distribution values from a permutation test (e.g., linear model which tests, for example, the influence of an adhd score and alcohol consumption on a stress (...
0 votes
0 answers
17 views

Bootstrapping for groups where within each group there are subgroups sampled unevenly

Say I collect two samples from two groups but the sample I collect from each subject within each group is a different size. ...
  • 1
1 vote
1 answer
43 views

Not sure on how to interpret the p-value for a permutation test

I was running a permutation test on my data. My goal was to test difference in means in two samples, by resampling one of them. I got results like these: And From my understanding, the p-value here ...
  • 13
0 votes
0 answers
24 views

How to compare observed versus expected ratio data between two independent, unpaired groups with different sample sizes?

so I'm comparing the expected fitness of double mutant animals (based on independent, additive effects of each corresponding single mutant) to the observed fitness of the double mutant. The data is ...
0 votes
1 answer
58 views

$R^2$ model comparison in test data

I want to compare two models of the form: ...
0 votes
0 answers
7 views

Metric to analyze shuffling on compression

I wish to analyze effect on shuffling on compression. Is there any other metric other than compression ratio and entropy? The metric should coorelate shuffling with a compression ratio.
2 votes
1 answer
57 views

CLT for t-test with unequal sample size; one group < 30?

CrossValidated has many discussions on how unequal variances are not a practical issue for two-sample t-tests when Welch correction is used and on how normality assumptions do not play a role (in Type-...
  • 7,947
0 votes
0 answers
13 views

Statistical test for low treatment sample size, high nb of record and large control group

I am looking to analyse the efficiency of a treatment. The problem is that I have only 3 patients in the treatment group. However: I have thousand of measurement of the value of interest for these 3 ...
  • 1,489
1 vote
0 answers
47 views

How to conduct a boostrapped or a permutation wilcoxon test (in R)

I have a total of 18 mice, 9 from Healthy group and 9 from Sick group. Since I have a very poor sample size I would like to use boostrapping or permutation test. Is there a way to conduct a ...
  • 467
1 vote
1 answer
57 views

Calculate a permutation-based p-value for a risk ratio

Summary How do you calculate a two-sided p-value for a risk ratio/relative risk (obtained from a GEE logistic regression via predicting risks with and without a treatment) based on a permutation test, ...
1 vote
1 answer
23 views

Is the expected value of a correlation in a randomly permuted sample zero?

Let's say we have two real valued data sets $x$ and $y$, both of length $n$. I do not want to make any further assumptions regarding these data sets. We're interested in their correlation. For testing ...
1 vote
1 answer
47 views

How to calculate p-value from the expected correlation coefficient using randomisation in R?

Here, I want to estimate the p-value using randomization. Tests of correlations involving patterns like Y (a function of X) vs. X involve a shared term and the X term. Here I want to randomize (999 ...
1 vote
0 answers
31 views

Monty Hall Gone Crazy [closed]

Can Someone help me out with these ?Answer along with solution will be much appreciated Q1) Imagine that now host have the following instructions. Put a prize behind a random door. Let the guest guess ...
0 votes
0 answers
20 views

Determine how likely a combination of two counts is compared to null models for both counts

I have two zero-adjusted negative binomial regression models fitted via gamlss not necessarily using the same covariates but at paired locations. ...
  • 33
1 vote
1 answer
64 views

Permutation testing: Which variable should be shuffled?

When conducting a permutation test, which variable should be shuffled? Let's say I have two variables X and Y, and my test statistic is the correlation coefficient between the two variables. To test ...
  • 21
1 vote
1 answer
38 views

Constrained ordination (CAP) pairwise ANOVA

Is there any way to perform pairwise ANOVA test from Constrained ordination for multiple factors in a variable? e.g. I am trying to figure out whether microbial communities among five plant genotypes ...
  • 41
1 vote
0 answers
13 views

How can I efficiently test whether a subsets of nodes in a DAG are "lined up" more often than expected by random chance?

I have a directed acyclic graph with N nodes, each of which is assigned to one of K groups, with K < N. My hypothesis is that nodes in the same group tend to "line up" along a linear path....
0 votes
0 answers
20 views

Permutation test replications aggregation

Short version: Is it right, and if yes, what is the right way to combine permutation tests applied on n randomly selected subsets of a bigger dataset? I was not able to find a publication that tackles ...
0 votes
0 answers
53 views

Conducting a Permutation test in R

I have a 200 × 102 data matrix of measurements of gene activity in male cancer patients (cases, last 52 columns) and healthy men (controls, first 50 columns). I want to do a permutation test to ...
  • 41
0 votes
0 answers
100 views

Permutation test involving multiple treatment groups where the test statistic is the precision of a single group

In a setting where we conduct multiple experiments in parallel, I am struggling to figure out whether reporting the significance of the precision of the measurements from a single experiment is a ...
  • 143
0 votes
0 answers
31 views

Replicability metric using a permutation test

I have been using a metric that is (effectively, under the hood) running a set of permutations tests, and then reporting a summary. I'm trying to unpack this metric and struggling to figure out if it ...
  • 143
0 votes
0 answers
25 views

I need guidance regarding a permutation based distribution problem

I am conducting a semester project that involves evaluating proportions of landcover (from NLCD) within animal survey areas (1km grid cells). A professor suggested that I randomly sample proportion of ...
0 votes
1 answer
53 views

Permutation test with covariates

I would like to perform a permutation test on a dataset with several batches and would like to take those batches into account during testing. Similar to a linear model ...
0 votes
0 answers
32 views

Careful comparison of strength of evidence (p-values)

Please hear me out before you roll your eyes when reading about "p-value comparisons". I have the following scenario: High-dimensional, clustered data and two groups within each cluster. ...
  • 53
0 votes
0 answers
65 views

Statistical significance and angular comparison (similarity test) between two vectors, MCMC and resampling approaches

I want to compare statistically if two vectors of PC1 loadings differ in terms of their direction which can be measured by the angle between them (e.g. this CV thread). I would like if someone can ...
  • 377
0 votes
1 answer
35 views

How to run a paired permutation test for 1 classifier, with 2 sets of predictions (paired), with classification accuracies as the test statistic?

CORE QUESTION: I have a single trained classifier (single participant) tested on 2 related multiclass classification tasks. As each trial of the classification tasks are related, the 2 sets of ...
2 votes
1 answer
72 views

How does violation of normality conditions affect p-values (relative to permutation tests)

I am running some t-tests using data from small samples (n=25) whose distributions do not appear to be normally distributed (they are not extremely skewed, but the skew is apparent in the plots and ...
0 votes
2 answers
56 views

Disagreement regarding the role of permutation/randomization/shuffling in permuation testing between two populations

A colleague and I recently got into a conversation/disagreement regarding permutation testing (e.g. for comparing the mean value of two populations). While i thought i had a fairly good understanding ...
0 votes
0 answers
88 views

Which statistical test should I use in my problem? (Permutation, Bootstrap, or some other test)

I have a dataset like this: H_0: The Test values are randomly generated. H_A: The Test values are related to the True values. ...
0 votes
0 answers
19 views

Cluster permutation test explained in simple terms

I am working with a set of EEG time-frequency data (ie. activity across time) that requires some stats to identify time windows where activity surpasses a threshold. One such test that could be used ...
1 vote
1 answer
1k views

How to interpret the feature importances for 'eli5.show_weights()' for regression?

I am trying to understand how the interpret the values yielded by eli5's show_weights variable after feature importance. I have used this for several regression models, e.g. multiple linear regression,...
2 votes
0 answers
94 views

How to specify my permutation scheme?

I am studying the biofilm microbiome of mountain lakes and plan to use PERMANOVA with adonis2 to assess the importance of some grouping variables (e.g. lakes with introduced fish vs. lakes without ...
0 votes
0 answers
20 views

Randomization-based test for comparisons of an observed proportion against an expected distribution

I generated a dataset of organisms and their gene organization, divided into groups of severely unequal sizes. My goal is to statistically compare each group's proportion of unique gene organizations ...
0 votes
2 answers
132 views

In permutation test, why do we take the proportion of sampled permutations with value equal or larger than the observed value?

The tutorial I followed explains permutation testing in an intuitive way. However, it has confused me in one specific part. Why do we take as p-value the proportion/probability of permutation with ...
  • 101
1 vote
0 answers
57 views

how do I perform permutation testinging for a prediction model developed within caret package (R)?

I'm fairly new to data science/StackExchange, so please excuse any faux pas I'm trying to perform permutation testing for a chosen ML algorithm (an elastic-net logistic regression) to derive a p-value....
  • 21
0 votes
1 answer
90 views

What is a "permutation" in PERMANOVA?

I'm using PERMANOVA to calculate significant differences between groups in my beta-diversity analysis for 16S rRNA gene sequencing. However, while I understand the results, I don't understand what the ...
1 vote
0 answers
18 views

How to correctly apply a PERMANOVA to a half-crossed experimental design

I'm a biologist and conducted an experiment in which I took analytical samples of aphid infested plants and analyzed them using metabolomics. The plants were chosen based on their Genotype (...
  • 21
0 votes
0 answers
45 views

Combining two levels of confidence interval/uncertainty

I am doing a random permutation test to estimate the confidence interval for a given statistic based on usual estimates of the Bernoulli distribution. I.e. assuming a normal approximation to the ...
1 vote
1 answer
72 views

Non-inferiority test for Spearman correlation

Let's say I have the following data and want to test whether the Spearman's correlation is > 0.5 ...
  • 398
2 votes
1 answer
228 views

Distance between two categorical distributions

I want to test whether two (empirical) categorical distributions taking on $K$ possible values (e.g. 5, with no innate underlying ordering) with associated (empirical) probabilities $p_k$ are the same....
  • 67
2 votes
1 answer
264 views

Permutation test vs Mann–Whitney U test

I am wondering if there exist any advantage of using the Mann-Whitney U test over a permutation test in the following setting: Suppose I have a group of animals with eggs in their brood chamber, from ...
  • 35

1
2 3 4 5
10