Stack Exchange Network

Stack Exchange network consists of 174 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange
Join us in building a kind, collaborative learning community via our updated Code of Conduct.

Signals situations where one is concerned about achieving intended power and size when more than one hypothesis test is performed.

0
votes
0answers
6 views

Multivariate hypothesis testing for proportions

It's very common in website optimisation to run A/B tests for conversion rates (e.g. 'buy' button is green by default, would the conversion rate from 'visit' to 'buy' be significantly larger if we ...
0
votes
0answers
8 views

What is the difference between seemingly unrelated regression (SUR) and correcting a set of OLS results for multiple comparisons?

As I understand it, the the seemingly unrelated regressions (SUR) or seemingly unrelated regression equations (SURE) models estimate a set of Ordinary Least Squares (OLS) equations where the error ...
0
votes
0answers
24 views

Bonferroni correction for pairwise proportion comparisons for more than one outcome level

Suppose I have an event that can have 3 different outcomes. This event is recorded for 21 different factor levels for a total of 100 times, and each time an outcome happens it is counted (so I am ...
1
vote
0answers
29 views

Comparing predictiveness of alternative class definitions using linear regression

I am comparing highly-correlated ways to classify patients as sick (1) or well (0). I want to see whether the way you define sickness impacts how well sickness predicts clinical outcomes. I have ...
0
votes
0answers
5 views

multiple comparisons on lmer tests using multcomp::glht() or running a number of models with different reference levels?

I am running a linear mixed effects model with an interaction between two variables. The interaction was significant. Now I am using multcomp::glht() to run planned comparisons (such as here[How to ...
4
votes
1answer
61 views

When testing multiple hypotheses, what does it mean when there are not enough extremes? [closed]

Suppose you are testing a large number of hypotheses, say a million. Unlike the usual situation where you have a lot of very small p-values, in this case all of your p-values are greater than 5%. ...
0
votes
0answers
37 views

Chi Square test for lot of categorical variables

I have 36 categorical variables, 11 binary and other with more than 2 levels. I wanted to check if they are independent or not, so I want to do Chi-square test of independence. Problem is I believe ...
0
votes
0answers
12 views

MANOVA assumption violation

This is about my PhD thesis where I am trying to see the pattern of memory deficit among different types of hypothyroidism. I have used a between subject group design. I have 1 IV with three levels. I ...
0
votes
0answers
20 views

Compare 2 classifiers on 2 datasets using F1-score

I am doing a little research in the field of multiclass imbalanced data classification. It entails comparing multiple "base" classifiers on 2 datasets. I followed the well known paper by Demsar and ...
1
vote
0answers
7 views

Chisquare for homogenity on subsets of samples for more detailed statements?

I have two samples of six treatment groups. The outcome is an amount of positives. The treatments in both samples are the same, the measured outcome is different but dichotomous in both samples. I ...
0
votes
0answers
8 views

Should I correct for multiple comparisons after simple main effects?

I had recently a discussion with a colleague and I can't figure out who is right. Suppose you run a study with two factors: A (level 1 and level 2) and B (Level 1, level 2 and level 3). You analyse ...
1
vote
1answer
19 views

GLMM summary and multcomp differences

Thanks you for you time to read this question. I run a GLMM because I have measure repeated on 100 individuals. The explanatory variable it is a proporcion so I use binomial family. The final model ...
2
votes
1answer
51 views

My deep learning model improved - is the difference significant?

Often deep learning/machine learning models are compared just by measuring their performance on a validation dataset - the model that performs better "wins". Within null hypothesis testing, two ...
0
votes
0answers
14 views

Most appropriate statistical test for multiple comparisons with categorical response variable in r

I am looking to compare the call type usage of humpback whales across three different group compositions. There are 13 different call types, and I would like to develop a model that will allow me to ...
0
votes
0answers
29 views

Adjusting groups of correlated tests for multiple testing in mediation models

I am interested in adjusting p values in the final outcome model of mediation analysis. I have looked up online and discussion here provides ample resources, however, I can't think of ways of directly ...
4
votes
0answers
48 views

Do results from the lme-function require adjustment of p-values?

If I run a linear mixed model with the lme() function and get results like these (comparing the score of 4 treatment groups against a placebo group): ...
0
votes
0answers
7 views

Multiple hypothesis testing: continuous vs. discrete treatments

I'm studying the effect of rainfall, cloud cover, and temperature on mood. I also want to examine whether these weather variables interact with each other. These are three separate (continuous) ...
2
votes
0answers
20 views

Are p-value corrections needed for separate tests with various grouping variables?

I have a dataset (find an example below) in which there are many factor variables (group_a, group_b, ...
-1
votes
0answers
9 views

Comparing multiple contingency tables for diagnostic tests

I have a set of subjects suspected of having a disease by a clinical examination. I take a biopsy that can be analyzed by 3 different lab tests. Only subjects suspected of having the disease will ...
1
vote
1answer
18 views

How to test for pairwise differences between factor levels while looking at an interaction with another factor?

Let's say I have a survey with questions 1 through 10 about different characteristics of animals rated from 1 to 10 and this is repeated for 5 different animals. Suppose I collect 50 surveys so my ...
3
votes
0answers
31 views

Is 'Significance test for Lasso' justified?

Lockhart et al described a "Significance test for the lasso", the motivating example for which is forward variable selection in a linear regression. In their Figures 1 and 2 they show Q-Q plots of a ...
0
votes
0answers
31 views

Data Mining and Multiple Testing

I have a dataset A which consist of 100000 data points. I am trying to find hidden relations in the data. I am following the steps below: Step 1. I partition data in 5 different random sets (20000 ...
3
votes
1answer
146 views

Statistical testing for cohort analysis and retention: did I do this right?

A common problem in tech-startups is cohort analysis when A/B testing new app features: within a certain time period between two different dates, two different sets of users joining between those ...
0
votes
0answers
28 views

Multiple comparison correction, when?

I am not sure about how many test I have in the following scenario: I collect 3 measures of a persons performance in a task (m1, m2, m3). I also collect data on two anatomical features (A1, A2). Then ...
-1
votes
0answers
20 views

Comparison of non-parametric tests

What is the difference between Dunn's test of multiple comparisons, the Mann-Whitney U test, the Wilcoxon rank sum test, and chisq.post.hoc from the R package ...
0
votes
0answers
32 views

Dunn's test of multiple comparisons

From the documentation page for the R function dunnTest from the package FSA: Performs Dunn's (1964) test of multiple ...
0
votes
3answers
28 views

Too many blood tests

In medicine, lab results are usually "normal" if within two standard deviations. If I were to order 20 random tests, would it be safe to assume that one "normal" result could fall outside the "normal"...
4
votes
1answer
141 views

Pairwise comparisons with emmeans for a mixed three-way interaction in a linear mixed-effects model

I have a rookie question about emmeans in R. I fit a complex model using lmer() with the following variables: A: a binary categorical predictor, within-subject B: a binary categorical predictor, ...
1
vote
1answer
23 views

How can I run multiple tests in a valid and random fashion?

I hope this is the correct place to ask this question. I'm currently doing research where I'm comparing two groups on their EEG frequency spectra. I wasn't too sure how to approach this. Data: I ...
0
votes
0answers
17 views

Testing 3 Kruskal Wallis on 3 datasets that each has 3 treatments

I am not that experienced, but, I read a lot and seems am stuck to know if what i am doing is right or not. I have 3 datasets that each were initiated based on session duration. The 1st dataset has ...
0
votes
0answers
11 views

opposite of multiple test correction

Is there a p value correction for dealing with multiple hypothesis testing when I want to know how many tests satisfy p > alpha (ie, how many tests fail to come up with a 'significant' result) instead ...
1
vote
0answers
20 views

Kolmogorov-Smirnov test for the distribution of p-values

I have a set of independent null hypothesis significance tests and their resulting p-values and I would like to make the claim that they are not all null by looking at the distribution of independent ...
0
votes
0answers
10 views

modified False Discovery Rate

In "Panning for Gold: Model-X Knockoffs for High-dimensional Controlled Variable Selection" by Candes et al, 2016, the authors discuss two notions of false discovery rate. Throughout, $q$ is the ...
0
votes
0answers
10 views

Comparing multiple independent distributions from two data sets

I am trying to find the best way to quatitatively compare two data sets. I would like to compare the width as well as the centroid of the distributions from the two data sets. Since there is many ...
1
vote
0answers
25 views

Removing non-significant main effects / interactions

I have a hypothesis that my dependent variable (a clinical outcome) is mediated both by main effects and by interactions between them. I want particularly to demonstrate that (or measure whether) the ...
1
vote
0answers
32 views

adjusted p-values for post-hoc tests?

I'm analysing about 20 lab-results of three different groups either with ANOVA, Kruskal–Wallis test or chi-squared. I correct the resulting p-values for multiple testing using the Benjamini/Hochberg ...
2
votes
0answers
15 views

Which test to use on partially overlapping groups

I have data in 4 categories, but they are not completely independent. To explain the data: I have looked at 40 Intelligent Tutoring Systems (ITSs) and determined which approach they use (model tracing,...
0
votes
0answers
26 views

Bonferroni/Holm correction when performing chi-squared residual analysis

Let's say I am performing two t-tests and one chi-squared test for a 5*2 table (for example levels of education vs. sex). I am further interested in which categories of education differ among sexes, ...
0
votes
0answers
25 views

Disparity between ANOVA results and pairwise comparisons (both heteroscedastic-consistent)

I have a data set consisting of a numerical response variable (y), and two factors (A with 3 levels, and ...
3
votes
1answer
30 views

FDR controlling demonstration in R

I'm trying to demonstrate BH method for controlling false discovery rate in R. The problem is that I get quite unexpected results (so I assume that something is going wrong with this) ...
0
votes
0answers
26 views

Cross-validation, feature subset and multiple testing correction

I have a binary prediction problem for which I am doing a 10 times 10-fold cross validation. Imagine that I do the CV analysis multiple times, each time selecting a different non-overlapping subset ...
2
votes
1answer
79 views

Testing for significant difference between coefficients from 2 different IV regressions (Stata or in general)

I'm running two different IV models in Stata that share the same DV and controls but have different instruments for the independent variable of interest. ...
0
votes
1answer
39 views

What are the (practical) differences between Tukey's HSD and the Dunn-Sidak correction procedures?

I have a data set with three groups, one of which is a control and the other two groups have been administered either a moderate or a high dose of a drug. The data are not normally or log-normally ...
1
vote
1answer
18 views

What qualifies as a set of independent tests to apply p-value adjustments?

There are many questions and answers on how to apply p-value adjustments and in which cases, but I couldn't get an answer to my problem based on those, so here goes : Assume you have a dataset with 5 ...
0
votes
0answers
42 views

Tukey's alternatives!

I have 10 independent groups and I need to know which one is significantly different than others. I am planning to use One-way ANOVA then multi-comparison test. I used Tukey's test but it always ...
2
votes
0answers
29 views

GAM - Smooth comparison with factor variables

I try to compare the splines for my 6 categories. To this end, first I tried the ordered factor approach and posted the result here. This comparison is not sufficient for me since I would like to have ...
0
votes
0answers
12 views

Are non-parametric test effective when my pairwise-evaluation data is multiplied through different distributions?

So, I've designed an experiment where I compare pairs of texts that have been clustered. I want to test a few different aspects of the data so I've created 8 different distributions and selected 100 ...
1
vote
0answers
16 views

How to adjust for the number of nested model tests?

How should I adjust for the number of tests I do when using the likelihood ratio difference test in a model estimated using maximum likelihood (or some flavor of), where the null hypothesis of the ...
-1
votes
1answer
19 views

How can I check if three or more results are statistically identical?

I conducted several experiments (for the moment 3, but there will be more in the near future), where in each experiment I made several measurements. I know (from previous experience) that the result ...
3
votes
1answer
52 views

Am I Double-Dipping on my Hypothesis Test?

I am testing a large number of simultaneous hypotheses (~300), with each hypothesis test trying to determine whether a given test statistic differs significantly from zero. Am I cheating the multiple ...