Questions tagged [permutation-test]

Statistical tests based on rearrangements of data that are consistent with the null hypothesis.

Filter by
Sorted by
Tagged with
0 votes
0 answers
16 views

Determine how likely a combination of two counts is compared to null models for both counts

I have two zero-adjusted negative binomial regression models fitted via gamlss not necessarily using the same covariates but at paired locations. ...
user avatar
  • 23
1 vote
1 answer
26 views

Permutation testing: Which variable should be shuffled?

When conducting a permutation test, which variable should be shuffled? Let's say I have two variables X and Y, and my test statistic is the correlation coefficient between the two variables. To test ...
user avatar
  • 21
0 votes
1 answer
26 views

Constrained ordination (CAP) pairwise ANOVA

Is there any way to perform pairwise ANOVA test from Constrained ordination for multiple factors in a variable? e.g. I am trying to figure out whether microbial communities among five plant genotypes ...
user avatar
  • 31
1 vote
0 answers
9 views

How can I efficiently test whether a subsets of nodes in a DAG are "lined up" more often than expected by random chance?

I have a directed acyclic graph with N nodes, each of which is assigned to one of K groups, with K < N. My hypothesis is that nodes in the same group tend to "line up" along a linear path....
user avatar
0 votes
0 answers
20 views

Permutation test replications aggregation

Short version: Is it right, and if yes, what is the right way to combine permutation tests applied on n randomly selected subsets of a bigger dataset? I was not able to find a publication that tackles ...
user avatar
0 votes
0 answers
32 views

Conducting a Permutation test in R

I have a 200 × 102 data matrix of measurements of gene activity in male cancer patients (cases, last 52 columns) and healthy men (controls, first 50 columns). I want to do a permutation test to ...
user avatar
  • 1
0 votes
0 answers
53 views

Permutation test involving multiple treatment groups where the test statistic is the precision of a single group

In a setting where we conduct multiple experiments in parallel, I am struggling to figure out whether reporting the significance of the precision of the measurements from a single experiment is a ...
user avatar
  • 143
0 votes
0 answers
26 views

Replicability metric using a permutation test

I have been using a metric that is (effectively, under the hood) running a set of permutations tests, and then reporting a summary. I'm trying to unpack this metric and struggling to figure out if it ...
user avatar
  • 143
0 votes
0 answers
21 views

I need guidance regarding a permutation based distribution problem

I am conducting a semester project that involves evaluating proportions of landcover (from NLCD) within animal survey areas (1km grid cells). A professor suggested that I randomly sample proportion of ...
user avatar
0 votes
1 answer
26 views

Permutation test with covariates

I would like to perform a permutation test on a dataset with several batches and would like to take those batches into account during testing. Similar to a linear model ...
user avatar
0 votes
0 answers
25 views

Careful comparison of strength of evidence (p-values)

Please hear me out before you roll your eyes when reading about "p-value comparisons". I have the following scenario: High-dimensional, clustered data and two groups within each cluster. ...
user avatar
  • 53
0 votes
0 answers
42 views

Statistical significance and angular comparison (similarity test) between two vectors, MCMC and resampling approaches

I want to compare statistically if two vectors of PC1 loadings differ in terms of their direction which can be measured by the angle between them (e.g. this CV thread). I would like if someone can ...
user avatar
  • 377
0 votes
1 answer
28 views

How to run a paired permutation test for 1 classifier, with 2 sets of predictions (paired), with classification accuracies as the test statistic?

CORE QUESTION: I have a single trained classifier (single participant) tested on 2 related multiclass classification tasks. As each trial of the classification tasks are related, the 2 sets of ...
user avatar
2 votes
1 answer
59 views

How does violation of normality conditions affect p-values (relative to permutation tests)

I am running some t-tests using data from small samples (n=25) whose distributions do not appear to be normally distributed (they are not extremely skewed, but the skew is apparent in the plots and ...
user avatar
0 votes
2 answers
41 views

Disagreement regarding the role of permutation/randomization/shuffling in permuation testing between two populations

A colleague and I recently got into a conversation/disagreement regarding permutation testing (e.g. for comparing the mean value of two populations). While i thought i had a fairly good understanding ...
user avatar
0 votes
0 answers
83 views

Which statistical test should I use in my problem? (Permutation, Bootstrap, or some other test)

I have a dataset like this: H_0: The Test values are randomly generated. H_A: The Test values are related to the True values. ...
user avatar
0 votes
0 answers
10 views

Cluster permutation test explained in simple terms

I am working with a set of EEG time-frequency data (ie. activity across time) that requires some stats to identify time windows where activity surpasses a threshold. One such test that could be used ...
user avatar
1 vote
1 answer
618 views

How to interpret the feature importances for 'eli5.show_weights()' for regression?

I am trying to understand how the interpret the values yielded by eli5's show_weights variable after feature importance. I have used this for several regression models, e.g. multiple linear regression,...
user avatar
2 votes
0 answers
49 views

How to specify my permutation scheme?

I am studying the biofilm microbiome of mountain lakes and plan to use PERMANOVA with adonis2 to assess the importance of some grouping variables (e.g. lakes with introduced fish vs. lakes without ...
user avatar
0 votes
0 answers
20 views

Randomization-based test for comparisons of an observed proportion against an expected distribution

I generated a dataset of organisms and their gene organization, divided into groups of severely unequal sizes. My goal is to statistically compare each group's proportion of unique gene organizations ...
user avatar
0 votes
2 answers
57 views

In permutation test, why do we take the proportion of sampled permutations with value equal or larger than the observed value?

The tutorial I followed explains permutation testing in an intuitive way. However, it has confused me in one specific part. Why do we take as p-value the proportion/probability of permutation with ...
user avatar
  • 101
1 vote
0 answers
44 views

how do I perform permutation testinging for a prediction model developed within caret package (R)?

I'm fairly new to data science/StackExchange, so please excuse any faux pas I'm trying to perform permutation testing for a chosen ML algorithm (an elastic-net logistic regression) to derive a p-value....
user avatar
  • 21
0 votes
1 answer
73 views

What is a "permutation" in PERMANOVA?

I'm using PERMANOVA to calculate significant differences between groups in my beta-diversity analysis for 16S rRNA gene sequencing. However, while I understand the results, I don't understand what the ...
user avatar
1 vote
0 answers
10 views

How to correctly apply a PERMANOVA to a half-crossed experimental design

I'm a biologist and conducted an experiment in which I took analytical samples of aphid infested plants and analyzed them using metabolomics. The plants were chosen based on their Genotype (...
user avatar
  • 21
0 votes
0 answers
35 views

Combining two levels of confidence interval/uncertainty

I am doing a random permutation test to estimate the confidence interval for a given statistic based on usual estimates of the Bernoulli distribution. I.e. assuming a normal approximation to the ...
user avatar
1 vote
1 answer
66 views

Non-inferiority test for Spearman correlation

Let's say I have the following data and want to test whether the Spearman's correlation is > 0.5 ...
user avatar
  • 388
1 vote
1 answer
70 views

Distance between two categorical distributions

I want to test whether two (empirical) categorical distributions taking on $K$ possible values (e.g. 5, with no innate underlying ordering) with associated (empirical) probabilities $p_k$ are the same....
user avatar
  • 57
2 votes
1 answer
149 views

Permutation test vs Mann–Whitney U test

I am wondering if there exist any advantage of using the Mann-Whitney U test over a permutation test in the following setting: Suppose I have a group of animals with eggs in their brood chamber, from ...
user avatar
  • 35
0 votes
1 answer
125 views

Permutation test on dependent data

Say that I have a sample of observations which are pairs of real numbers $(x_i, y_i)$. Pairs are i.i.d., but $x_i$ and $y_i$ are not independent. I want to test whether the mean of $x$ is equal to the ...
user avatar
0 votes
0 answers
26 views

When does the 'equal' affect most in the permutation test?

First of all, I need to use a permutation test to calculate the significance of some data. When I do the permutation test, I need to shuffle the data and compare the original one and the shuffled one. ...
user avatar
  • 101
0 votes
0 answers
43 views

Appropriate test of conditional difference in means

I have some quantity $X \in \mathbb{R}$, and labels $A \in \{0, 1\}$ and $B \in \{0, 1\}$ which can be thought of as a labelling of overlapping groups (i.e., an observation belongs to group $A$ if $A =...
user avatar
2 votes
1 answer
29 views

Is the FDR linear-step up procedure Benjamini and Hochberg valid for empircal p-values from non-parametric tests (e.g., bootstrap)?

I have empirical p-values from bootstrapped hypothesis tests with 10,000 resamples each. Because there are a finite number of resamples, some of the p-values are "0", i.e., < 0.0001 with ...
user avatar
  • 51
1 vote
1 answer
51 views

Consistency of Wilcoxon rank sum statistic

In Chapter 1 of E. L. Lehmann's book Nonparametrics, he refers to the Wilcoxon Rank Sum test in treatment-control experiments. Using Lehmann's notation, let $N$ be the total number of units, $n$ the ...
user avatar
6 votes
2 answers
227 views

Permutation testing for machine learning: permute entire set or only training set?

Ojala and Garriga (2010) [Journal of Machine Learning Research 11 (2010) 1833-1863] mention two permutation tests for machine learning: test one for assessing whether a classifier has learned anything ...
user avatar
  • 165
0 votes
0 answers
15 views

Difference between Mantel test and Quadratic Assignment Procedure

Could someone clearly define the difference between the Mantel test and the Quadratic Assignment Procedure (QAP)? I am needing to examine whether an asymmetrical matrix is dependent upon a symmetrical ...
user avatar
0 votes
0 answers
29 views

Permuting data to seperate marginal and interaction effects

I've read a bit on permutation tests for discovering interactions. Are there permutations I can perform on my dataset to try to separate the interactions from the marginals? In particular, I want to ...
user avatar
  • 23
0 votes
0 answers
72 views

maxT permutation hypothesis test for correlated test

I understand the concept and procedure of the maxT permutation test to control for the FWER when using multiple comparisons. I read that this approach is also suitable to correct for multiple ...
user avatar
  • 13
0 votes
1 answer
319 views

Investigate significance of adonis (Permanova) coefficients

Adonis' output for an analysis of multivariate variation between groups includes coefficients for all variables of the dataset. Is anyone aware of a way to ...
user avatar
  • 109
0 votes
0 answers
91 views

How to test the difference between dependent partial correlations in the presence of autocorrelation?

Background and what I am trying to do: I have a single dependent variable $Y$, and $K$ independent variables of interest ($X_1, X_2... X_k$). I also have a single independent variable of non-interest, ...
user avatar
1 vote
1 answer
65 views

permutation test on correlation coefficient

I have a correlation coefficient of two hierarchical clustering trees with 20 labels. I want to check the hypothesis that the labels of the trees are uncorrelated with a permutation test. I calculated ...
user avatar
  • 35
0 votes
0 answers
13 views

Am I inputting the correct data for randomisation tests for my single-case design in R?

I'm currently doing randomisation tests (RTs) for a single-case design in R. A bit of context about my phases: I am doing an AB design, which was staggered using a multiple baseline design across 5 ...
user avatar
2 votes
1 answer
63 views

Permutation statistics yield different results when computed again

I am doing a within-subject design experiment and trying to analyze my data. To simplify my problem, let's say that there is condition A and condition B, and I have 24 pariticipants. I want to do ...
user avatar
  • 23
0 votes
1 answer
37 views

Intuition behind permutations + probability

I'm new to stats so I'm struggling to grasp the intuition behind permutations + probability. Would anybody be able to help me with parts (b) and (c) of this question? Any help is greatly appreciated :)...
user avatar
  • 3
0 votes
1 answer
74 views

Doubts with P-value and Hypothesis in Permutation Test problem

I have to solve this problem for a stats course: An e-commerce company is testing a new design for a web page. The objective is to achieve at least a 2% increase in the conversion rate. An experiment ...
user avatar
  • 1
1 vote
1 answer
552 views

Permutation test and wilcoxon rank-sum test giving different results

I am comparing the revenue (at user level) between control & test to see if there is a significant difference. Both control & test have roughly about 39k data points . I am doing a permutation ...
user avatar
  • 33
0 votes
0 answers
27 views

Statistical test for assessing trends (one-sample)

I'm interested in looking at annual trends. The numbers represent the total number of a particular animal species that have been counted in that year. We see a decreasing trend, and I would like to ...
user avatar
  • 1
1 vote
0 answers
7 views

Compare the performance of a machine learning informed classification regime against a random one

I am building an active machine learning toolkit to speed up articles' classification for systematic reviews. This method has three steps: Order records heuristically according to some relevant term ...
user avatar
  • 2,313
0 votes
1 answer
117 views

Analyzing replicates of contingency tables

Problem statement We would like to measure the effect of treatment on cell differentiation, in other words whether or not specific treatments cause cells to change their cell types. So we treat cells, ...
user avatar
1 vote
0 answers
25 views

Random forest on compound analysis and input data permutation

I am using a random forest model to associate climate variables with a specific type of impact, which is measured as the likelihood of failure (therefore, classification). The choice of random forest ...
user avatar
  • 127
3 votes
1 answer
339 views

Adjusting P values for multiple comparisons using permutation tests

I have a number of continuous predictors (biomarker measurements) which I would like to test for association with a binary outcome variable (disease status), adjusting for multiple comparisons. As ...
user avatar
  • 31

1
2 3 4 5
9