Questions tagged [permutation-test]

Statistical tests based on rearrangements of data that are consistent with the null hypothesis.

Filter by
Sorted by
Tagged with
0
votes
0answers
4 views

In multivariate analysis using PERMANOVA and NMDS, is it appropriate to include both predictors in my model?

I am an undergrad student new to much of these statistical tests. I am using NMDS ordination and Permanova testing to analyze variation in cover type (using count data) at different distance classes (...
0
votes
0answers
4 views

Compare the performance of a machine learning informed classification regime against a random one

I am building an active machine learning toolkit to speed up articles' classification for systematic reviews. This method has three steps: Order records heuristically according to some relevant term ...
0
votes
1answer
22 views

Analyzing replicates of contingency tables

Problem statement We would like to measure the effect of treatment on cell differentiation, in other words whether or not specific treatments cause cells to change their cell types. So we treat cells, ...
0
votes
0answers
10 views

Random forest on compound analysis and input data permutation

I am using a random forest model to associate climate variables with a specific type of impact, which is measured as the likelihood of failure (therefore, classification). The choice of random forest ...
2
votes
1answer
43 views

Adjusting P values for multiple comparisons using permutation tests

I have a number of continuous predictors (biomarker measurements) which I would like to test for association with a binary outcome variable (disease status), adjusting for multiple comparisons. As ...
0
votes
0answers
24 views

What similarity index should I use with relative abundance data for a PERMANOVA?

I have data on the relative abundance (0 to 1 ratio) of 5 land covers for different landscapes (analogous to species composition). These landscapes are grouped into three geographic areas and I want ...
0
votes
0answers
22 views

PLS model performs significantly worse than chance according to permutation test

I'm using Partial Least Squares regression to relate two datasets: X has 40 variables, Y has 54 variables, both have 26 observations. Because I don't have a lot of observations, I'm using a ...
0
votes
0answers
22 views

Permutation test for non-normal distributions?

I am analyzing if the habitat type (treatment) may affect to the virus incidence. For that aim I performed a permutation analysis to this dataset: ...
0
votes
0answers
42 views

Can the Permutation Test be used for this hypothesis testing problem?

To investigate the association between two categorical variables, some statistical tests exist, for example, the chi-squared test. We didn't cover these test in the course. For example, suppose we ...
0
votes
0answers
18 views

R: vegan/permute: additional level of nesting for pseudoreplicates in a split plot

I am trying to investigate how to set up a permutation scheme for community data originating from a split plot experiment with pseudoreplicates at the lowest level. A theoretical design could be the ...
0
votes
0answers
79 views

how to interpret negative values in permutation importance report?

i am using a permutation report from sklearn. i train my model and optimize on 'mean_squared_error' this is actually negated in sklearn. when i get the perumutation importance report however how do i ...
3
votes
0answers
42 views

What model assumptions are necessary for a changepoint detection based on permutation test

I am writing about the change point detection algorithm suggested by Taylor W. in my bachelor thesis. The full description can be found here: https://variation.com/wp-content/uploads/change-point-...
3
votes
0answers
30 views

permutation test sampling

In statistical inference, specifically while conducting hypothesis testing, we use permutation test, is there any reason why resampling is done without replacement (shuffle), what difference does it ...
1
vote
0answers
31 views

Combining tests: Maximum test statistic method but one test is a permutation test

I want to combine two tests $A,B$ for the same null and alternative hypotheses. The motivation is that it has been shown that test $A$ has higher power in some situations whereas test $B$ has higher ...
0
votes
0answers
37 views

Implementation of a statistical test to compare the ratio of two means

I have the data (samples) that represent population sizes of different bacterial cultures from 4 distributions (I suspect they are all normal) for which I would like to compare the ratio of the means ...
0
votes
0answers
19 views

R: When to set bias.adjust = TRUE in vegan::betadisper?

I am trying to test if my data have homogeneous variance using vegan::betadisper() in R. I read the help for the function but it only says that is appropriate when ...
0
votes
1answer
24 views

Combining p-values from permutation tests from single subjects

I have a question that's very similar to this question: Combine multiple independent permutation tests My 'issue' is exactly the same: I have 20 subjects and I used permutation tests to determine the ...
0
votes
0answers
34 views

Permutation test - Median vs Mean

I am trying to study the influence of a Treatment Model (denoted by 1) vs a control group (denote by 0). The goal of my statistical analysis is to be able to say with a certain confidence that the ...
0
votes
0answers
26 views

Correct way to implement the H0 of no correlation between random variables X and Y with a permutation test?

I want to test whether a measurement error corrected correlation between variables X and Y is significantly different from 0 or not. I have 65 paired observations (x1,y1), (x2, y2),...(x65, y65). Due ...
0
votes
1answer
50 views

Bootstrap/Permutation test for equality of two distributions

In the book "An Introduction to the Bootstrap" the authors state that both the permutation test and the bootstrap test can be used to test $H_0:F=G$, where $F$ and $G$ are two different ...
2
votes
1answer
22 views

Does Variance Reduction really help the Nonparametric Test?

In Online Experiment, Working on Variance Reductions could help a lot for the traditional parametric tests like test two-proportion (CTR) or two-mean t-test, as it significantly improves the power and ...
0
votes
0answers
27 views

Does permutation test require iid?

Suppose I have a treatment group X and independent from control group Y. The data set x1,x2..., y1,y2... Let's say I want to test the mean of the difference If the randomization unit is a session ...
0
votes
0answers
17 views

Permutation testing and covariate/confound inclusion random statistics permutation

I have a data set with two groups where participants in both groups were collected from numerous sites. I performed randomised permutation testing between the groups. However, I was asked if I ...
0
votes
0answers
34 views

A (new?) bootstrap-based technique to check if a model performs worse on “subpopulation”

This is sort of an ad hoc statistical test I've devised to assess the performance of a prediction model on different subpopulations I have a large sample A with a Subgroup R. I would like to tell if ...
1
vote
1answer
362 views

Alternative to Bonferroni Correction when performing multiple one-vs-rest association tests

My question anonymizes the 'nouns' of the question to protect my employer. It is not really about lab rats and experimental treatments. I am also coming more from a Machine Learning background, so my ...
1
vote
0answers
57 views

Permutation test - can I apply it here?

I am struggling to understand if I can use a permutation test on my data. I have the following setup: A spine with three vertebral bodies (Level). Fat rings were added to the spine to simulate 5 ...
0
votes
0answers
82 views

Is it possible to use a PERMANOVA (adonis / R) on a time series - are there any alternatives?

I'm having trouble with fitting a PERMANOVA (adonis) to my data and am actually not sure if it is possible at all. I have a pitfall arthropod data set for 5 research areas and 3 consecutive field ...
0
votes
0answers
18 views

Permuting outcome variable to estimate null likelihood ratio distribution for Poisson GLM

I fit a Poisson GLM using a set of candidate explanatory variables ($\lambda = e^{\beta_{0} + \beta_{1}x_{1}+\beta_{2}x_{2}+...}$), tested against the global null hypothesis ($\lambda = e^{\beta_{0}}$)...
2
votes
1answer
32 views

Appropriate practice for setting seeds in permutation analyses?

I am currently using permutation ANCOVA for a study of mine, in R, using the permuco package. Running multiple models for different brain regions of interest (ROIs). What I am finding is that the p-...
0
votes
0answers
45 views

Permutation test for A/B testing

I have done an extensive survey of testing methodologies in A/B practices. It seemed that default non parametric test choice is MWW U test. However, permutation test is known to generally have a ...
0
votes
0answers
31 views

When conducting multiple comparisons with adonis, the p-values of the comparisons should be used as they are or used it after correction?

I recently ran 60 tests with adonis and found some significant differences (p-vaue < 0.05) between control and treatment (I only have one control and one treatment and do not need to do the ...
2
votes
1answer
47 views

Permutation Hypothesis Testing procedure

I'm not strong in statistics and I'm looking for a help. I work with real estate data and I want to compare apartment prices in 2 districts: district "A" and district "B". Data ...
0
votes
0answers
28 views

General procedures to check conditions for hypothesis testing?

I am going to have a linear by linear test(lbl_test in R, and the related doc can be found here) but the math underlying it is still a mystery to me after trying my best to comprehend this ...
0
votes
0answers
12 views

permutation based clustering for fNIRS data in R

I hope this finds the group safe and well. I am working on a problem involving clustering a series of channels. I have a fNIRS dataset, which is the average hemodynamic response across 49 channels ...
0
votes
1answer
29 views

Implement cluster-based permutation test for regression

I have worked with permutation testing previously to assess significance in classification problems. What you commonly do here is to randomly swap group labels to generate a null distribution. However,...
2
votes
1answer
31 views

Getting feature weights with permutation_test_score()

I am using sklearn to fit a SVM to some data. Since I wanted to use cross-validation and evaluate my classification accuracy using permutations, I am using the permutation_test_score() function (https:...
3
votes
0answers
130 views

Which test should I use to compare the effect of a parasite infection on gut microbiome across time serie data point

I am interested in the effect of an intestinal parasite infection on the rat gut bacterial community. I have two groups of rats ("infected" and "non-infected") each consists of 6 ...
1
vote
0answers
60 views

Accounting for spatial structure in constrained ordination analysis (vegan)

I am looking to use ordination to test whether certain environmental measures influence the microbial community in soil samples. I am a bit confused about the correct way to define the model and ...
0
votes
0answers
99 views

Appropriate effect size to report for permutation tests (Anova specifically)

Original question, with explanation of outcome and explanatory variables included. Suppose I have a neuroimaging dataset, e.g. diffusion tensor-derived measure for multiple white matter bundles (or ...
0
votes
1answer
145 views

How to evaluate permutation tests in vegan when strata are are not balanced

I have a multivariate data set of 12 sites, and 4 plots per site, except one site which only has three plots. There two variables of interest: site_type, which is ...
0
votes
0answers
79 views

Permutation Testing Machine Learning Model Performance With Caret in R

Using the caret package in R, classification machine learning model performance is usually assessed using the confusionmatrix() function, which uses a Binomial Test ...
0
votes
0answers
89 views

vegan::adonis2() for randomized complete block design with main-plot, split-plot, and nested effects

I coded my adonis2 formula following the answer below. The original r-forge is not accessible anymore, though. I used the suggested model on my data but I got the ...
0
votes
1answer
54 views

How to Properly Interpret the Hypotheses/Results of a Permutation Test Based on the Test Statistic?

I asked a question recently: Is This an Appropriate Application of a Permutation Test?. All information about my data, the permutation test and the results can be found there, but here is a quick ...
2
votes
0answers
289 views

What is the difference between Permutation Importance and Drop Column Importance?

I would like to understand what is the difference between Permutation Importance (as outlined by Breiman in his original paper on Random Forests) and Drop Column Importance. From a cursory look at ...
5
votes
1answer
98 views

Is this an Appropriate Application of a Permutation Test?

Suppose I have count data between three groups, each with a different number of observations: $n_1 = 11, n_2 = 6$ and $n_3 = 5$. My data is listed below: \begin{array} {|r|r|}\hline group 1 & ...
0
votes
0answers
222 views

How does adonis2 (Permanova) handle continuous variables?

I'm working with a matrix of community count data. We collected both continuous and categorical environmental information.I would like to use adonis2 (R package vegan) to apply permutational analysis ...
2
votes
0answers
22 views

Likelihood as a test statistic in a hypothesis test

Suppose I have two samples $S_1,S_2$ of categorical data, and I'd like to design a hypothesis test to check the null hypothesis that they are both iid samples from the same underlying distribution. ...
1
vote
0answers
18 views

Inconsistent results from partial Mantel test on (non)distance matrices

Currently I am trying to look at the correlation between three matrices of ecological data. All three are forms of distance matrices (two matrices of pairwise community dissimilarities and one matrix ...
1
vote
1answer
60 views

When you do a random permutation F test (by permuting group membership) is inference made on the samples or the populations?

I am trying to understand different approaches that use randomization procedures. One thing I cannot come to a clear conclusion on is this: Say we have randomly sampled individuals from 2 populations. ...
1
vote
0answers
26 views

Permutation testing for significance of a predictor

I've been thinking lately about connecting permutation tests with testing for significance of a variable. Consider the following (very simple) example: Suppose we want to consider the following linear ...

1
2 3 4 5
9