Questions tagged [permutation-test]

Statistical tests based on rearrangements of data that are consistent with the null hypothesis.

Filter by
Sorted by
Tagged with
0
votes
2answers
25 views

In permutation test, why do we take the proportion of sampled permutations with value equal or larger than the observed value?

The tutorial I followed explains permutation testing in an intuitive way. However, it has confused me in one specific part. Why do we take as p-value the proportion/probability of permutation with ...
1
vote
0answers
8 views

how do I perform permutation testinging for a prediction model developed within caret package (R)?

I'm fairly new to data science/StackExchange, so please excuse any faux pas I'm trying to perform permutation testing for a chosen ML algorithm (an elastic-net logistic regression) to derive a p-value....
0
votes
1answer
38 views

What is a "permutation" in PERMANOVA?

I'm using PERMANOVA to calculate significant differences between groups in my beta-diversity analysis for 16S rRNA gene sequencing. However, while I understand the results, I don't understand what the ...
0
votes
0answers
6 views

How to correctly apply a PERMANOVA to a half-crossed experimental design

I'm a biologist and conducted an experiment in which I took analytical samples of aphid infested plants and analyzed them using metabolomics. The plants were chosen based on their Genotype (...
0
votes
0answers
22 views

Combining two levels of confidence interval/uncertainty

I am doing a random permutation test to estimate the confidence interval for a given statistic based on usual estimates of the Bernoulli distribution. I.e. assuming a normal approximation to the ...
1
vote
1answer
48 views

Non-inferiority test for Spearman correlation

Let's say I have the following data and want to test whether the Spearman's correlation is > 0.5 ...
1
vote
1answer
45 views

Distance between two categorical distributions

I want to test whether two (empirical) categorical distributions taking on $K$ possible values (e.g. 5, with no innate underlying ordering) with associated (empirical) probabilities $p_k$ are the same....
2
votes
1answer
51 views

Permutation test vs Mann–Whitney U test

I am wondering if there exist any advantage of using the Mann-Whitney U test over a permutation test in the following setting: Suppose I have a group of animals with eggs in their brood chamber, from ...
0
votes
1answer
36 views

Permutation test on dependent data

Say that I have a sample of observations which are pairs of real numbers $(x_i, y_i)$. Pairs are i.i.d., but $x_i$ and $y_i$ are not independent. I want to test whether the mean of $x$ is equal to the ...
0
votes
0answers
23 views

When does the 'equal' affect most in the permutation test?

First of all, I need to use a permutation test to calculate the significance of some data. When I do the permutation test, I need to shuffle the data and compare the original one and the shuffled one. ...
0
votes
0answers
23 views

Appropriate test of conditional difference in means

I have some quantity $X \in \mathbb{R}$, and labels $A \in \{0, 1\}$ and $B \in \{0, 1\}$ which can be thought of as a labelling of overlapping groups (i.e., an observation belongs to group $A$ if $A =...
2
votes
1answer
21 views

Is the FDR linear-step up procedure Benjamini and Hochberg valid for empircal p-values from non-parametric tests (e.g., bootstrap)?

I have empirical p-values from bootstrapped hypothesis tests with 10,000 resamples each. Because there are a finite number of resamples, some of the p-values are "0", i.e., < 0.0001 with ...
1
vote
1answer
40 views

Consistency of Wilcoxon rank sum statistic

In Chapter 1 of E. L. Lehmann's book Nonparametrics, he refers to the Wilcoxon Rank Sum test in treatment-control experiments. Using Lehmann's notation, let $N$ be the total number of units, $n$ the ...
5
votes
2answers
92 views

Permutation testing for machine learning: permute entire set or only training set?

Ojala and Garriga (2010) [Journal of Machine Learning Research 11 (2010) 1833-1863] mention two permutation tests for machine learning: test one for assessing whether a classifier has learned anything ...
0
votes
0answers
9 views

Difference between Mantel test and Quadratic Assignment Procedure

Could someone clearly define the difference between the Mantel test and the Quadratic Assignment Procedure (QAP)? I am needing to examine whether an asymmetrical matrix is dependent upon a symmetrical ...
0
votes
0answers
28 views

Permuting data to seperate marginal and interaction effects

I've read a bit on permutation tests for discovering interactions. Are there permutations I can perform on my dataset to try to separate the interactions from the marginals? In particular, I want to ...
0
votes
0answers
37 views

maxT permutation hypothesis test for correlated test

I understand the concept and procedure of the maxT permutation test to control for the FWER when using multiple comparisons. I read that this approach is also suitable to correct for multiple ...
0
votes
1answer
132 views

Investigate significance of adonis (Permanova) coefficients

Adonis' output for an analysis of multivariate variation between groups includes coefficients for all variables of the dataset. Is anyone aware of a way to ...
0
votes
0answers
61 views

How to test the difference between dependent partial correlations in the presence of autocorrelation?

Background and what I am trying to do: I have a single dependent variable $Y$, and $K$ independent variables of interest ($X_1, X_2... X_k$). I also have a single independent variable of non-interest, ...
0
votes
0answers
35 views

To do a test via bootstrap, should I bootstrap the actual data the shifted (under the null) data or the test statistics?

Please help me to resolve the source my confusion. Let us assume I want to do a test via bootstrap. For simplicity, let it be comparison of two means. I saw the following approaches: to resample &...
1
vote
1answer
43 views

permutation test on correlation coefficient

I have a correlation coefficient of two hierarchical clustering trees with 20 labels. I want to check the hypothesis that the labels of the trees are uncorrelated with a permutation test. I calculated ...
0
votes
0answers
12 views

Am I inputting the correct data for randomisation tests for my single-case design in R?

I'm currently doing randomisation tests (RTs) for a single-case design in R. A bit of context about my phases: I am doing an AB design, which was staggered using a multiple baseline design across 5 ...
2
votes
1answer
53 views

Permutation statistics yield different results when computed again

I am doing a within-subject design experiment and trying to analyze my data. To simplify my problem, let's say that there is condition A and condition B, and I have 24 pariticipants. I want to do ...
0
votes
1answer
31 views

Intuition behind permutations + probability

I'm new to stats so I'm struggling to grasp the intuition behind permutations + probability. Would anybody be able to help me with parts (b) and (c) of this question? Any help is greatly appreciated :)...
0
votes
0answers
29 views

compute likelihood statistic with permutation test

This question is about hypothesis testing, where we want to use the likelihood ratio statistic with permutations test. Suppose we sample $n$ observations from the distribution $F_{XY}$, which is the ...
0
votes
1answer
54 views

Doubts with P-value and Hypothesis in Permutation Test problem

I have to solve this problem for a stats course: An e-commerce company is testing a new design for a web page. The objective is to achieve at least a 2% increase in the conversion rate. An experiment ...
0
votes
0answers
42 views

Implementing permutation tests in randomized controlled trials with repeated measures

I am trying to implement a permutation test to use for testing treatment effects in a repeated measures RCT-design. Unfortunately I am having some trouble understanding how to design a correct ...
0
votes
0answers
31 views

Alternatives of permutation test to proof that a model performs better than chance level

I built a model based on a very high dimensional data, which requires 4-5 days to built using High Performance Computing (HPC). I want to do a statistical test to proof that the predictive power of my ...
0
votes
0answers
4 views

Difference between PERMANOVA and Multiple Regression on distance Matrices

I am a newbie in statistics so the math part of these 2 tests might not be crystal clear. But it seems that they are quite similar if you try to analyze e.g. how 4 variables explain the dissimilarity ...
0
votes
1answer
234 views

Permutation test and wilcoxon rank-sum test giving different results

I am comparing the revenue (at user level) between control & test to see if there is a significant difference. Both control & test have roughly about 39k data points . I am doing a permutation ...
0
votes
0answers
22 views

Statistical test for assessing trends (one-sample)

I'm interested in looking at annual trends. The numbers represent the total number of a particular animal species that have been counted in that year. We see a decreasing trend, and I would like to ...
0
votes
0answers
110 views

In multivariate analysis using PERMANOVA and NMDS, is it appropriate to include both predictors in my model?

I am an undergrad student new to much of these statistical tests. I am using NMDS ordination and Permanova testing to analyze variation in cover type (using count data) at different distance classes (...
0
votes
0answers
6 views

Compare the performance of a machine learning informed classification regime against a random one

I am building an active machine learning toolkit to speed up articles' classification for systematic reviews. This method has three steps: Order records heuristically according to some relevant term ...
0
votes
1answer
70 views

Analyzing replicates of contingency tables

Problem statement We would like to measure the effect of treatment on cell differentiation, in other words whether or not specific treatments cause cells to change their cell types. So we treat cells, ...
1
vote
0answers
18 views

Random forest on compound analysis and input data permutation

I am using a random forest model to associate climate variables with a specific type of impact, which is measured as the likelihood of failure (therefore, classification). The choice of random forest ...
3
votes
1answer
186 views

Adjusting P values for multiple comparisons using permutation tests

I have a number of continuous predictors (biomarker measurements) which I would like to test for association with a binary outcome variable (disease status), adjusting for multiple comparisons. As ...
0
votes
0answers
59 views

What similarity index should I use with relative abundance data for a PERMANOVA?

I have data on the relative abundance (0 to 1 ratio) of 5 land covers for different landscapes (analogous to species composition). These landscapes are grouped into three geographic areas and I want ...
0
votes
0answers
51 views

PLS model performs significantly worse than chance according to permutation test

I'm using Partial Least Squares regression to relate two datasets: X has 40 variables, Y has 54 variables, both have 26 observations. Because I don't have a lot of observations, I'm using a ...
0
votes
0answers
31 views

Permutation test for non-normal distributions?

I am analyzing if the habitat type (treatment) may affect to the virus incidence. For that aim I performed a permutation analysis to this dataset: ...
0
votes
0answers
85 views

Can the Permutation Test be used for this hypothesis testing problem?

To investigate the association between two categorical variables, some statistical tests exist, for example, the chi-squared test. We didn't cover these test in the course. For example, suppose we ...
0
votes
0answers
35 views

R: vegan/permute: additional level of nesting for pseudoreplicates in a split plot

I am trying to investigate how to set up a permutation scheme for community data originating from a split plot experiment with pseudoreplicates at the lowest level. A theoretical design could be the ...
3
votes
0answers
55 views

What model assumptions are necessary for a changepoint detection based on permutation test

I am writing about the change point detection algorithm suggested by Taylor W. in my bachelor thesis. The full description can be found here: https://variation.com/wp-content/uploads/change-point-...
3
votes
1answer
66 views

permutation test sampling

In statistical inference, specifically while conducting hypothesis testing, we use permutation test, is there any reason why resampling is done without replacement (shuffle), what difference does it ...
1
vote
0answers
40 views

Combining tests: Maximum test statistic method but one test is a permutation test

I want to combine two tests $A,B$ for the same null and alternative hypotheses. The motivation is that it has been shown that test $A$ has higher power in some situations whereas test $B$ has higher ...
0
votes
1answer
72 views

Combining p-values from permutation tests from single subjects

I have a question that's very similar to this question: Combine multiple independent permutation tests My 'issue' is exactly the same: I have 20 subjects and I used permutation tests to determine the ...
0
votes
0answers
82 views

Permutation test - Median vs Mean

I am trying to study the influence of a Treatment Model (denoted by 1) vs a control group (denote by 0). The goal of my statistical analysis is to be able to say with a certain confidence that the ...
0
votes
1answer
98 views

Bootstrap/Permutation test for equality of two distributions

In the book "An Introduction to the Bootstrap" the authors state that both the permutation test and the bootstrap test can be used to test $H_0:F=G$, where $F$ and $G$ are two different ...
2
votes
1answer
26 views

Does Variance Reduction really help the Nonparametric Test?

In Online Experiment, Working on Variance Reductions could help a lot for the traditional parametric tests like test two-proportion (CTR) or two-mean t-test, as it significantly improves the power and ...
0
votes
0answers
91 views

Does permutation test require iid?

Suppose I have a treatment group X and independent from control group Y. The data set x1,x2..., y1,y2... Let's say I want to test the mean of the difference If the randomization unit is a session ...
2
votes
1answer
1k views

Alternative to Bonferroni Correction when performing multiple one-vs-rest association tests

My question anonymizes the 'nouns' of the question to protect my employer. It is not really about lab rats and experimental treatments. I am also coming more from a Machine Learning background, so my ...

1
2 3 4 5
9