Questions tagged [permutation-test]

Statistical tests based on rearrangements of data that are consistent with the null hypothesis.

Filter by
Sorted by
Tagged with
2
votes
1answer
19 views

Does Variance Reduction really help the Nonparametric Test?

In Online Experiment, Working on Variance Reductions could help a lot for the traditional parametric tests like test two-proportion (CTR) or two-mean t-test, as it significantly improves the power and ...
-1
votes
0answers
12 views

How many ways 5’ end of CRISPR-Cas9 sequence be modified? 5’ end has gRNA with a length of 20 nucleotides [closed]

Clue: Order matters and repetitions are allowed 20^4 20!/16! 20!/16!x4! 4^20
0
votes
0answers
36 views

Resampling hypothesis tests: why does permutation test have lower power than bootstrap?

I am working with an unusual summary of some data and I want to test whether the summary from group A is significantly different from that for group B. I can capture uncertainty about the summary ...
0
votes
0answers
20 views

Does permutation test require iid?

Suppose I have a treatment group X and independent from control group Y. The data set x1,x2..., y1,y2... Let's say I want to test the mean of the difference If the randomization unit is a session ...
0
votes
0answers
17 views

Permutation testing and covariate/confound inclusion random statistics permutation

I have a data set with two groups where participants in both groups were collected from numerous sites. I performed randomised permutation testing between the groups. However, I was asked if I ...
0
votes
0answers
31 views

A (new?) bootstrap-based technique to check if a model performs worse on “subpopulation”

This is sort of an ad hoc statistical test I've devised to assess the performance of a prediction model on different subpopulations I have a large sample A with a Subgroup R. I would like to tell if ...
1
vote
1answer
38 views

Alternative to Bonferroni Correction when performing multiple one-vs-rest association tests

My question anonymizes the 'nouns' of the question to protect my employer. It is not really about lab rats and experimental treatments. I am also coming more from a Machine Learning background, so my ...
1
vote
0answers
50 views

Permutation test - can I apply it here?

I am struggling to understand if I can use a permutation test on my data. I have the following setup: A spine with three vertebral bodies (Level). Fat rings were added to the spine to simulate 5 ...
0
votes
0answers
19 views

Is it possible to use a PERMANOVA (adonis / R) on a time series - are there any alternatives?

I'm having trouble with fitting a PERMANOVA (adonis) to my data and am actually not sure if it is possible at all. I have a pitfall arthropod data set for 5 research areas and 3 consecutive field ...
0
votes
0answers
16 views

Permuting outcome variable to estimate null likelihood ratio distribution for Poisson GLM

I fit a Poisson GLM using a set of candidate explanatory variables ($\lambda = e^{\beta_{0} + \beta_{1}x_{1}+\beta_{2}x_{2}+...}$), tested against the global null hypothesis ($\lambda = e^{\beta_{0}}$)...
2
votes
1answer
28 views

Appropriate practice for setting seeds in permutation analyses?

I am currently using permutation ANCOVA for a study of mine, in R, using the permuco package. Running multiple models for different brain regions of interest (ROIs). What I am finding is that the p-...
0
votes
0answers
25 views

Permutation test for A/B testing

I have done an extensive survey of testing methodologies in A/B practices. It seemed that default non parametric test choice is MWW U test. However, permutation test is known to generally have a ...
0
votes
0answers
17 views

When conducting multiple comparisons with adonis, the p-values of the comparisons should be used as they are or used it after correction?

I recently ran 60 tests with adonis and found some significant differences (p-vaue < 0.05) between control and treatment (I only have one control and one treatment and do not need to do the ...
2
votes
1answer
37 views

Permutation Hypothesis Testing procedure

I'm not strong in statistics and I'm looking for a help. I work with real estate data and I want to compare apartment prices in 2 districts: district "A" and district "B". Data ...
0
votes
0answers
25 views

General procedures to check conditions for hypothesis testing?

I am going to have a linear by linear test(lbl_test in R, and the related doc can be found here) but the math underlying it is still a mystery to me after trying my best to comprehend this ...
0
votes
0answers
8 views

permutation based clustering for fNIRS data in R

I hope this finds the group safe and well. I am working on a problem involving clustering a series of channels. I have a fNIRS dataset, which is the average hemodynamic response across 49 channels ...
0
votes
0answers
22 views

Implement cluster-based permutation test for regression

I have worked with permutation testing previously to assess significance in classification problems. What you commonly do here is to randomly swap group labels to generate a null distribution. However,...
2
votes
1answer
21 views

Getting feature weights with permutation_test_score()

I am using sklearn to fit a SVM to some data. Since I wanted to use cross-validation and evaluate my classification accuracy using permutations, I am using the permutation_test_score() function (https:...
3
votes
0answers
61 views

Which test should I use to compare the effect of a parasite infection on gut microbiome across time serie data point

I am interested in the effect of an intestinal parasite infection on the rat gut bacterial community. I have two groups of rats ("infected" and "non-infected") each consists of 6 ...
1
vote
0answers
32 views

Accounting for spatial structure in constrained ordination analysis (vegan)

I am looking to use ordination to test whether certain environmental measures influence the microbial community in soil samples. I am a bit confused about the correct way to define the model and ...
0
votes
0answers
35 views

Appropriate effect size to report for permutation tests (Anova specifically)

Original question, with explanation of outcome and explanatory variables included. Suppose I have a neuroimaging dataset, e.g. diffusion tensor-derived measure for multiple white matter bundles (or ...
0
votes
1answer
63 views

How to evaluate permutation tests in vegan when strata are are not balanced

I have a multivariate data set of 12 sites, and 4 plots per site, except one site which only has three plots. There two variables of interest: site_type, which is ...
0
votes
0answers
46 views

Permutation Testing Machine Learning Model Performance With Caret in R

Using the caret package in R, classification machine learning model performance is usually assessed using the confusionmatrix() function, which uses a Binomial Test ...
0
votes
0answers
23 views

vegan::adonis2() for randomized complete block design with main-plot, split-plot, and nested effects

I coded my adonis2 formula following the answer below. The original r-forge is not accessible anymore, though. I used the suggested model on my data but I got the ...
0
votes
1answer
39 views

How to Properly Interpret the Hypotheses/Results of a Permutation Test Based on the Test Statistic?

I asked a question recently: Is This an Appropriate Application of a Permutation Test?. All information about my data, the permutation test and the results can be found there, but here is a quick ...
2
votes
0answers
125 views

What is the difference between Permutation Importance and Drop Column Importance?

I would like to understand what is the difference between Permutation Importance (as outlined by Breiman in his original paper on Random Forests) and Drop Column Importance. From a cursory look at ...
5
votes
1answer
65 views

Is this an Appropriate Application of a Permutation Test?

Suppose I have count data between three groups, each with a different number of observations: $n_1 = 11, n_2 = 6$ and $n_3 = 5$. My data is listed below: \begin{array} {|r|r|}\hline group 1 & ...
0
votes
0answers
51 views

How does adonis2 (Permanova) handle continuous variables?

I'm working with a matrix of community count data. We collected both continuous and categorical environmental information.I would like to use adonis2 (R package vegan) to apply permutational analysis ...
2
votes
0answers
19 views

Likelihood as a test statistic in a hypothesis test

Suppose I have two samples $S_1,S_2$ of categorical data, and I'd like to design a hypothesis test to check the null hypothesis that they are both iid samples from the same underlying distribution. ...
1
vote
0answers
11 views

Inconsistent results from partial Mantel test on (non)distance matrices

Currently I am trying to look at the correlation between three matrices of ecological data. All three are forms of distance matrices (two matrices of pairwise community dissimilarities and one matrix ...
0
votes
1answer
29 views

When you do a random permutation F test (by permuting group membership) is inference made on the samples or the populations?

I am trying to understand different approaches that use randomization procedures. One thing I cannot come to a clear conclusion on is this: Say we have randomly sampled individuals from 2 populations. ...
1
vote
0answers
20 views

Permutation testing for significance of a predictor

I've been thinking lately about connecting permutation tests with testing for significance of a variable. Consider the following (very simple) example: Suppose we want to consider the following linear ...
1
vote
0answers
7 views

How to statistically analyse gene-environment interactions pairs?

I am new here so sorry I hope I am using the forum correctly. Also, my knowledge on statistics is very basic and is something I am working towards learning more. I am studying gene-environment ...
0
votes
0answers
15 views

Hypothesis Testing Distributions after Permutations

I have collected data on an event taking place and have obtained a set of true observations, say X samples. I am trying to determine whether these observations are different from what they should be, ...
0
votes
0answers
19 views

p value for permutation test for interaction data

I have a data for pair-wise interactions between genes. The genes can have 4 possible states (Hypo / Hyper-methylated and Up/Down-regulated). I have taken the top 1000 interactions (ordered based on ...
0
votes
0answers
34 views

Permutation test in R multiple p values

I've got this data: ...
0
votes
1answer
29 views

What is a permutation test?

I am trying to understand what a permutation test in a perMANOVA is. Can someone either explain how data is being substituted for each permutation or direct me to a good visualization on youtube?
0
votes
0answers
12 views

Which is the best approach for robustness testing for deep learning models?

I tried 10 fold CV and my validation accuracy is above 95%. Just to check the robustness of my CNN model I wanted to do a permutation testing. But is that necessary?
0
votes
1answer
35 views

Which is the best approach for permutation testing for deep learning models? [closed]

Should I just shuffle my labels and train my model for a certain number of iterations? The result of permutation testing should give low testing/validation accuracy, right? Is it necessary to do a ...
1
vote
0answers
166 views

adonis vs. adonis2

I am trying to conduct permanova tests on a dataset. There are multiple options for doing this in R and I am not sure how to decide what to use. What is the difference between adonis() and adonis2() ...
0
votes
0answers
38 views

When distributions are non normal and heteroscedastic, is it preferred to use ordinal logistic regression or permutation tests?

I am currently conducting statistical tests on my two independent samples (both with more than 1500 entries each). The sample sizes are no equal. My response variables are interval as well as quasi ...
0
votes
0answers
17 views

Interpreting results of ANOSIM conducted on non-abundance data

I am trying to understand how to implement and interpret an analysis of similarities (using the vegan package in R) in the following situation. Consider the following data frame: ...
0
votes
2answers
73 views

'one-way anova' randomization test in R

I want to implement basic randomization tests as described in Ernst, 2004 Permutation methods for exact inference. I have observations of a single continuous variable for n individuals falling into k ...
0
votes
0answers
24 views

Pearson correlation not distributed around zero when using nonnegative least squares (NNLS) in permutation test?

I am trying to explain artificially created fmri data in MATLAB [1] through a linear combination of w neural network layers. To compare those, I transformed the data into respresentational ...
3
votes
1answer
43 views

Selection of test statistic for permutation test of variability

I have two arrays, $x_1$ and $x_2$ and want to see if there is a statistically significant difference in variance between them. A permutation test makes sense, where the test statistic would be the ...
1
vote
0answers
29 views

Can a permutation test be used on time series to quantify an effect of the marketing campaign?

I am trying to accept or reject a hypothesis that a marketing campaign had positive effect on sales of some product. I am working with time series of monthly sales. The series is not stationary (it ...
7
votes
3answers
289 views

Permutation tests and exchangeability [duplicate]

Permutation tests assume exchangeability of the response/observations under the null hypothesis. In what practical situations is this clearly violated? When is it unproblematic? Edit/additional ...
2
votes
1answer
46 views

Clarification of methods for using non-parametric tests to compare groups- vegan package

I would like to validate my approach to comparing groups using non-parametric tests. data created below describes the structure of my data. Each ...
1
vote
0answers
13 views

Control of type I error rate in functional ANOVA F-test

I am following how to compare curves via functional data analysis, specifically functional ANOVA. This video makes sense in describing it as being a lot like vanilla ANOVA...up to a point. When we do ...
0
votes
0answers
16 views

Dealing with duplicated values in points in pearson correlation testing

I am looking to analyze the correlation between the body weights of people and their body fat percentages. There are two difficulties in my data: the first is that I have multiple data points per ...

1
2 3 4 5
9