# Questions tagged [permutation-test]

Statistical tests based on rearrangements of data that are consistent with the null hypothesis.

441 questions
Filter by
Sorted by
Tagged with
16 views

### Determine how likely a combination of two counts is compared to null models for both counts

I have two zero-adjusted negative binomial regression models fitted via gamlss not necessarily using the same covariates but at paired locations. ...
• 23
1 vote
26 views

### Permutation testing: Which variable should be shuffled?

When conducting a permutation test, which variable should be shuffled? Let's say I have two variables X and Y, and my test statistic is the correlation coefficient between the two variables. To test ...
• 21
26 views

### Constrained ordination (CAP) pairwise ANOVA

Is there any way to perform pairwise ANOVA test from Constrained ordination for multiple factors in a variable? e.g. I am trying to figure out whether microbial communities among five plant genotypes ...
• 31
1 vote
9 views

### How can I efficiently test whether a subsets of nodes in a DAG are "lined up" more often than expected by random chance?

I have a directed acyclic graph with N nodes, each of which is assigned to one of K groups, with K < N. My hypothesis is that nodes in the same group tend to "line up" along a linear path....
20 views

### Permutation test replications aggregation

Short version: Is it right, and if yes, what is the right way to combine permutation tests applied on n randomly selected subsets of a bigger dataset? I was not able to find a publication that tackles ...
32 views

### Conducting a Permutation test in R

I have a 200 × 102 data matrix of measurements of gene activity in male cancer patients (cases, last 52 columns) and healthy men (controls, first 50 columns). I want to do a permutation test to ...
53 views

### Permutation test involving multiple treatment groups where the test statistic is the precision of a single group

In a setting where we conduct multiple experiments in parallel, I am struggling to figure out whether reporting the significance of the precision of the measurements from a single experiment is a ...
• 143
26 views

### Replicability metric using a permutation test

I have been using a metric that is (effectively, under the hood) running a set of permutations tests, and then reporting a summary. I'm trying to unpack this metric and struggling to figure out if it ...
• 143
21 views

### I need guidance regarding a permutation based distribution problem

I am conducting a semester project that involves evaluating proportions of landcover (from NLCD) within animal survey areas (1km grid cells). A professor suggested that I randomly sample proportion of ...
26 views

### Permutation test with covariates

I would like to perform a permutation test on a dataset with several batches and would like to take those batches into account during testing. Similar to a linear model ...
• 115
25 views

### Careful comparison of strength of evidence (p-values)

Please hear me out before you roll your eyes when reading about "p-value comparisons". I have the following scenario: High-dimensional, clustered data and two groups within each cluster. ...
• 53
42 views

### Statistical significance and angular comparison (similarity test) between two vectors, MCMC and resampling approaches

I want to compare statistically if two vectors of PC1 loadings differ in terms of their direction which can be measured by the angle between them (e.g. this CV thread). I would like if someone can ...
• 377
28 views

### How to run a paired permutation test for 1 classifier, with 2 sets of predictions (paired), with classification accuracies as the test statistic?

CORE QUESTION: I have a single trained classifier (single participant) tested on 2 related multiclass classification tasks. As each trial of the classification tasks are related, the 2 sets of ...
59 views

### How does violation of normality conditions affect p-values (relative to permutation tests)

I am running some t-tests using data from small samples (n=25) whose distributions do not appear to be normally distributed (they are not extremely skewed, but the skew is apparent in the plots and ...
41 views

### Disagreement regarding the role of permutation/randomization/shuffling in permuation testing between two populations

A colleague and I recently got into a conversation/disagreement regarding permutation testing (e.g. for comparing the mean value of two populations). While i thought i had a fairly good understanding ...
83 views

### Which statistical test should I use in my problem? (Permutation, Bootstrap, or some other test)

I have a dataset like this: H_0: The Test values are randomly generated. H_A: The Test values are related to the True values. ...
10 views

### Cluster permutation test explained in simple terms

I am working with a set of EEG time-frequency data (ie. activity across time) that requires some stats to identify time windows where activity surpasses a threshold. One such test that could be used ...
1 vote
618 views

### How to interpret the feature importances for 'eli5.show_weights()' for regression?

I am trying to understand how the interpret the values yielded by eli5's show_weights variable after feature importance. I have used this for several regression models, e.g. multiple linear regression,...
49 views

### How to specify my permutation scheme?

I am studying the biofilm microbiome of mountain lakes and plan to use PERMANOVA with adonis2 to assess the importance of some grouping variables (e.g. lakes with introduced fish vs. lakes without ...
20 views

### Randomization-based test for comparisons of an observed proportion against an expected distribution

I generated a dataset of organisms and their gene organization, divided into groups of severely unequal sizes. My goal is to statistically compare each group's proportion of unique gene organizations ...
57 views

### In permutation test, why do we take the proportion of sampled permutations with value equal or larger than the observed value?

The tutorial I followed explains permutation testing in an intuitive way. However, it has confused me in one specific part. Why do we take as p-value the proportion/probability of permutation with ...
• 101
1 vote
44 views

### how do I perform permutation testinging for a prediction model developed within caret package (R)?

I'm fairly new to data science/StackExchange, so please excuse any faux pas I'm trying to perform permutation testing for a chosen ML algorithm (an elastic-net logistic regression) to derive a p-value....
• 21
73 views

### What is a "permutation" in PERMANOVA?

I'm using PERMANOVA to calculate significant differences between groups in my beta-diversity analysis for 16S rRNA gene sequencing. However, while I understand the results, I don't understand what the ...
1 vote
10 views

### How to correctly apply a PERMANOVA to a half-crossed experimental design

I'm a biologist and conducted an experiment in which I took analytical samples of aphid infested plants and analyzed them using metabolomics. The plants were chosen based on their Genotype (...
• 21
35 views

### Combining two levels of confidence interval/uncertainty

I am doing a random permutation test to estimate the confidence interval for a given statistic based on usual estimates of the Bernoulli distribution. I.e. assuming a normal approximation to the ...
• 101
1 vote
66 views

### Non-inferiority test for Spearman correlation

Let's say I have the following data and want to test whether the Spearman's correlation is > 0.5 ...
• 388
1 vote
70 views

### Distance between two categorical distributions

I want to test whether two (empirical) categorical distributions taking on $K$ possible values (e.g. 5, with no innate underlying ordering) with associated (empirical) probabilities $p_k$ are the same....
• 57
149 views

### Permutation test vs Mann–Whitney U test

I am wondering if there exist any advantage of using the Mann-Whitney U test over a permutation test in the following setting: Suppose I have a group of animals with eggs in their brood chamber, from ...
• 35
125 views

### Permutation test on dependent data

Say that I have a sample of observations which are pairs of real numbers $(x_i, y_i)$. Pairs are i.i.d., but $x_i$ and $y_i$ are not independent. I want to test whether the mean of $x$ is equal to the ...
26 views

### When does the 'equal' affect most in the permutation test?

First of all, I need to use a permutation test to calculate the significance of some data. When I do the permutation test, I need to shuffle the data and compare the original one and the shuffled one. ...
• 101
43 views

1 vote
65 views

### permutation test on correlation coefficient

I have a correlation coefficient of two hierarchical clustering trees with 20 labels. I want to check the hypothesis that the labels of the trees are uncorrelated with a permutation test. I calculated ...
• 35
13 views

### Am I inputting the correct data for randomisation tests for my single-case design in R?

I'm currently doing randomisation tests (RTs) for a single-case design in R. A bit of context about my phases: I am doing an AB design, which was staggered using a multiple baseline design across 5 ...
63 views

### Permutation statistics yield different results when computed again

I am doing a within-subject design experiment and trying to analyze my data. To simplify my problem, let's say that there is condition A and condition B, and I have 24 pariticipants. I want to do ...
• 23
37 views

### Intuition behind permutations + probability

I'm new to stats so I'm struggling to grasp the intuition behind permutations + probability. Would anybody be able to help me with parts (b) and (c) of this question? Any help is greatly appreciated :)...
74 views

### Doubts with P-value and Hypothesis in Permutation Test problem

I have to solve this problem for a stats course: An e-commerce company is testing a new design for a web page. The objective is to achieve at least a 2% increase in the conversion rate. An experiment ...
• 1
1 vote
552 views

### Permutation test and wilcoxon rank-sum test giving different results

I am comparing the revenue (at user level) between control & test to see if there is a significant difference. Both control & test have roughly about 39k data points . I am doing a permutation ...
• 33
27 views

### Statistical test for assessing trends (one-sample)

I'm interested in looking at annual trends. The numbers represent the total number of a particular animal species that have been counted in that year. We see a decreasing trend, and I would like to ...
• 1
1 vote
7 views

### Compare the performance of a machine learning informed classification regime against a random one

I am building an active machine learning toolkit to speed up articles' classification for systematic reviews. This method has three steps: Order records heuristically according to some relevant term ...
• 2,313
117 views

### Analyzing replicates of contingency tables

Problem statement We would like to measure the effect of treatment on cell differentiation, in other words whether or not specific treatments cause cells to change their cell types. So we treat cells, ...
• 25
1 vote
25 views

### Random forest on compound analysis and input data permutation

I am using a random forest model to associate climate variables with a specific type of impact, which is measured as the likelihood of failure (therefore, classification). The choice of random forest ...
• 127