Stack Exchange Network

Stack Exchange network consists of 174 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Signals situations where one is concerned about achieving intended power and size when more than one hypothesis test is performed.

0
votes
0answers
17 views

Using test MSE to compare models

I was reading through the definition of MSE and the formula I have found in all articles is the one shown in https://en.wikipedia.org/wiki/Mean_squared_error , which is the expected squared difference ...
0
votes
1answer
14 views

Multiple comaprison and secondary research

It's maybe some kind of philosophical question. I have a large dataset with information about customers. I return to the dataset from time to time to make some estimations or check different ...
0
votes
1answer
21 views

How to obtain the results of the Nemenyi post-hoc test in a table showing grouped pairs?

I performed a Friedman's test, which showed that there are differences among my treatments. Thus, I choose to perform the Nemenyi's post-hoc test in order to infer which treatments statistically ...
0
votes
1answer
11 views

qPCR Gene expression validation and multiple testing

It's common practice to adjust for multiple testing when analyzing high-throughput gene expression data, eg. RNA-seq and microarrays. Despite this, researchers routinely select a subset of genes for ...
1
vote
1answer
22 views

emmeans pairwise contrasts result in same output values for all?

First of all, I am a beginner at mixed models, so I beg your patience and advice if this post could be improved. The structure of the data: ...
1
vote
2answers
38 views

$R^2$ and p-value for multivariate linear regression with confounders

I am working on a project where I am interested in the following variables: Dependent variables: $y_1,\, y_2,\, y_3,\, y_4,\, y_5$ (continuous) Independent variables: $x_1,\, x_2, \ldots,\, x_{18}$ ...
0
votes
0answers
16 views

How to compare 4 different time series and check for statistical significance among them?

I have a data comprising the proportion of a certain bat behavior across nights from 1800h to 1930h. I want to compare and check whether there is a difference in the behavior with 4 varying moon ...
1
vote
0answers
9 views

Indiscpecies - multipatt and overcoming multi-comparrisons

What is the best way to overcome the potential issue of type 1 error when doing an indicator analysis using the multipatt function in the ...
1
vote
0answers
23 views

comparison of two mean curves

I have two mean values based on the same sample of people, call them $x_1$ and $x_2$. I know that to compare two means, I would usually use a t-test. However, what I have is a plot of each against a ...
0
votes
0answers
8 views

significance test by confidence interval in complete randomization design(CRD)

In page 91 of this book, Design and analysis of experiments by Montgomery , the author stated after Equation (3.30) that: Clearly, if the confidence interval in Equation 3.30 includes zero, ...
0
votes
1answer
32 views

How can I test whether an individual treatment mean significant or not?

Suppose by complete randomization design, I reach into a decision that there is at least difference between two treatment means. That is, my hypothesis is $$H_o:\mu_1=\mu_2=\mu_3=\mu_4=\mu_5$$ $$...
0
votes
0answers
9 views

Multiple Comparisons, Risk Models

I am planning to do a survey which collects information on four predictors of crime using Likert scales along with demographics. Some of the demographics are categorical. The response variables are ...
0
votes
0answers
16 views

Is two-factor factorial design most appropriate in this situation?

Suppose there are two factors, A and B. There are 7 levels for factor A and 4 levels of ...
1
vote
1answer
17 views

Multiple Comparison Correct Group of Confidence Intervals

I have a number of data points (~10,000) compared against different confidence intervals to determine whether or not they are significantly elevated or not. Now I would like to correct for multiple ...
0
votes
0answers
12 views

Comparative Analysis

First I would like to indicate I plan to use Bayesian analysis methods (as instructed) I want to compare the effect of 6 independent variables (IVs) on four dependent variables. The dependent ...
0
votes
0answers
13 views

Multiple Comparison in non-parametric setting?

I have the following problem: There are 4 independent non-Gaussian continuous distributions, A, B, C, D, and I want to see whether one of A and B is "superior" to both of C and D. I'm aware of large ...
3
votes
1answer
129 views

How to interpret the result of Friedman's test?

I uploaded the data and then I used bootstrapping to have 10 different samples from the original data but with the same length as the original data. For each sample, I used 7 distance metrics, and I ...
2
votes
1answer
31 views

Type I and Type II errors when varying the number of treatments

I have a hypothetical sample of n observations, and I want to test if there are significant differences across multiple treatments. I need to know how the number of treatments interfere in the type I ...
0
votes
0answers
8 views

Multi-arm bandit Time Lag in the Update

Normally the multi-arm bandit problem relies on an instantaneous update on the success criteria (i.e. CTR either happens or it doesn't instantaneously). What does one do if there is a time lag in the ...
0
votes
1answer
24 views

Post hoc testing without initial ANOVA

I am comparing the response to a drug under five different conditions. The textbook instruction is to perform a one-way ANOVA followed by a post hoc test if necessary. The P value from an ANOVA is ...
0
votes
1answer
43 views

How to state my model in R in words (linear mixed effect model, general linear hypothesis, Tukey, Benjamini Hochberg)

I have a dataset that has four treatment groups (AX, AY, BX, BY). I am trying to determine if there is a significant difference in a response variable (Length) between any of these four treatment ...
2
votes
1answer
20 views

Which of these scenarios are multiple or repeated comparisons?

I believed I knew what multiple comparison or repeated testing meant but upon reading more on the subject and listening to others I have become more confused. My confusion started with A/B testing ...
1
vote
0answers
23 views

Which statistical test to determine interaction effect?

I conducted an experiment to determine the effect of blood type on blood cell interactions with proteins of differing blood groups and want to determine if there is an interaction effect for the ...
1
vote
0answers
8 views

How to demonstrate that a generation process outperforms some others, given a parameter?

Dear statistical stackexchanger, I would like to evaluate whether an algorithm outperforms others for a task of increasing and increasing complexity. I am measuring the impact of the selected ...
0
votes
0answers
16 views

Which 'single-step' multiple comparison test is being employed within R multcomp::glht?

When performing a general linear hypothesis test using the glht function within the R multcomp library. For example: ...
0
votes
0answers
11 views

Correcting p-values from jackknife resamples

I have data from few subjects (N=10), compare the DV using ANOVA and get a p-value. Since I have few subjects, I want to see if the significant p-value didn't just happen due to one subject. Ok, I ...
0
votes
0answers
16 views

Analogue of critical difference (CD) diagram for the comparison over single dataset

does anybody know any fancy way to present the performance of multiple classifiers (in terms of AUC, for example) over just 1, single dataset? I am familiar with the critical difference diagram for ...
1
vote
1answer
45 views

Formulae for FDR values [duplicate]

Can anybody give a formula for the "FDR values"? (I write "FDR values" because strictly speaking, you don't have an FDR value for each test, but an FDR value for the whole study, but nevertheles, you ...
0
votes
0answers
39 views

30 persons, 3 related readouts

I am stuck with how to best analyze the following experimental design: We analyze 3 variables (A, B, C), that are not independent from each other: if one of them is high, the other ones are expected ...
0
votes
0answers
131 views

P-value adjustment for hierarchical / multilevel data

I'm dealing with a large data set which is hierarchical in nature, that is 1000 schools, within every school 1000 students. The dependent variable is a measure of performance. Ideally I would like to ...
1
vote
1answer
15 views

Comparing same item but before and after result

I have a questionnaire for the effectiveness of the implementation of a short course. Respondents need to answer 6 items in part B of my questionnaire with 4-point likert scale (ordinal data). The ...
1
vote
1answer
43 views

2-way ANOVA multiple comparisons question

My question is about a 2-way ANOVA for an experiment. For simplicity let's say I'm using 3 methods to stimulate cells at 3 different time points, and I want to look at a resulting output variable. ...
1
vote
0answers
6 views

Kendalls u-coefficient to calculate overall consistency of pair comparison experiments

I have 8 conditions. In a pairwise experiment, I am comparing 28 pairs: condition1 vs condition2, condition3 vs. condition 4, condition1 vs condition3, condition4 vs, condition2, etc etc. Each ...
2
votes
1answer
198 views

Bonferroni correction: how does it work?

I already read the answers here: About the Bonferroni correction, but I still don't understand the concept completely. Assume the following hypothetical case, where I perform an ANOVA test to ...
0
votes
2answers
30 views

How to Control Familywise Error Rate in a series of Planned McNemar Comparisons

I have paired binary responses from same subjects under 9 different conditions. I conducted Cochran's Q which indicated that there significantly different pairs exist. I want to follow up but NOT ...
0
votes
1answer
54 views

Pairwise comparisons on mcglm joint model in r

Using the mcglm package in r, I have been able to get pairwise comparisons from single-DV models but have not figured out how to get pairwise comparisons for an IV according to both DVs in a joint ...
1
vote
1answer
22 views

Understanding Kenward-Rogers Degrees of Freedom from lsmeans()

I'm trying to analyze data using a multilevel model which predicts subject response times using the experimental group and trial validity. ...
0
votes
0answers
29 views

Correct approach to performing multiple comparisons?

Preamble I have conducted an experiment to determine whether my method is sensitive to receiving the true data or random data. In all of my experiments, I worked with a collection of 1,000 data ...
0
votes
0answers
28 views

Multivariate hypothesis testing for proportions

It's very common in website optimisation to run A/B tests for conversion rates (e.g. 'buy' button is green by default, would the conversion rate from 'visit' to 'buy' be significantly larger if we ...
0
votes
0answers
14 views

What is the difference between seemingly unrelated regression (SUR) and correcting a set of OLS results for multiple comparisons?

As I understand it, the the seemingly unrelated regressions (SUR) or seemingly unrelated regression equations (SURE) models estimate a set of Ordinary Least Squares (OLS) equations where the error ...
0
votes
0answers
53 views

Bonferroni correction for pairwise proportion comparisons for more than one outcome level

Suppose I have an event that can have 3 different outcomes. This event is recorded for 21 different factor levels for a total of 100 times, and each time an outcome happens it is counted (so I am ...
1
vote
0answers
29 views

Comparing predictiveness of alternative class definitions using linear regression

I am comparing highly-correlated ways to classify patients as sick (1) or well (0). I want to see whether the way you define sickness impacts how well sickness predicts clinical outcomes. I have ...
0
votes
0answers
12 views

multiple comparisons on lmer tests using multcomp::glht() or running a number of models with different reference levels?

I am running a linear mixed effects model with an interaction between two variables. The interaction was significant. Now I am using multcomp::glht() to run planned comparisons (such as here[How to ...
4
votes
1answer
69 views

When testing multiple hypotheses, what does it mean when there are not enough extremes? [closed]

Suppose you are testing a large number of hypotheses, say a million. Unlike the usual situation where you have a lot of very small p-values, in this case all of your p-values are greater than 5%. ...
0
votes
0answers
80 views

Chi Square test for lot of categorical variables

I have 36 categorical variables, 11 binary and other with more than 2 levels. I wanted to check if they are independent or not, so I want to do Chi-square test of independence. Problem is I believe ...
0
votes
0answers
21 views

MANOVA assumption violation

This is about my PhD thesis where I am trying to see the pattern of memory deficit among different types of hypothyroidism. I have used a between subject group design. I have 1 IV with three levels. I ...
0
votes
0answers
27 views

Compare 2 classifiers on 2 datasets using F1-score

I am doing a little research in the field of multiclass imbalanced data classification. It entails comparing multiple "base" classifiers on 2 datasets. I followed the well known paper by Demsar and ...
1
vote
0answers
7 views

Chisquare for homogenity on subsets of samples for more detailed statements?

I have two samples of six treatment groups. The outcome is an amount of positives. The treatments in both samples are the same, the measured outcome is different but dichotomous in both samples. I ...
0
votes
0answers
15 views

Should I correct for multiple comparisons after simple main effects?

I had recently a discussion with a colleague and I can't figure out who is right. Suppose you run a study with two factors: A (level 1 and level 2) and B (Level 1, level 2 and level 3). You analyse ...
1
vote
1answer
29 views

GLMM summary and multcomp differences

Thanks you for you time to read this question. I run a GLMM because I have measure repeated on 100 individuals. The explanatory variable it is a proporcion so I use binomial family. The final model ...