Signals situations where one is concerned about achieving intended power and size when more than one hypothesis test is performed.

learn more… | top users | synonyms (2)

0
votes
1answer
22 views

Permutation tests and multiple testing

I'm using permutation tests, based on random forest importance measures, to perform variables selection in my dataset. Using 99 permutations, my permutation tests Pvalues are comprised between 0.01 ...
2
votes
1answer
67 views

Hypothesis testing on Poisson Binomial distribution

Imagine $N$ baskets each filled with a different number of products chosen from a set $P$. It is possible to have repeated products in the same basket. For instance: ...
1
vote
1answer
33 views

Multiple testing problem - can these test results be caused by chance?

I conduct 30 independent significance tests on $p<.05$. I observe that in $X=3$ cases the test is rejected. Can this event be caused by chance due to a multiple testing problem? I understand that ...
0
votes
0answers
11 views

non significance with one-way ANOVA, but significance with multiple comparisons [duplicate]

I'm comparing deviation between three directions (vrt, lng and lat). I applied ...
1
vote
0answers
11 views

Controlling for Type 1 Errors: Post-hoc testing on more than 1 ANOVA

I have conducted several repeated-measures ANOVA's with post-hoc testing via Tukey's HSD whenever the omnibus F-test was significant. I'm having some confusion about controlling for type 1 errors when ...
0
votes
1answer
31 views

Grouping predictor factor levels based on response variable

I've read that it's bad to do this, but am looking for details as to why. Suppose we're trying to fit the linear model $Y_i = \beta_0 + \beta_1 X_{1i} + \beta_2 X_{2i} + \epsilon_i$ where $Y$ is ...
0
votes
1answer
92 views

Why is the Tukey confidence interval wider than the T-test confidence interval?

I have two confidence intervals for a comparison of two means. The first is a t-test CI and the other is a Tukey CI. The Tukey CI $(2.36, 26.5)$ is wider than the t-test CI $(6.46, 21.33)$. What is ...
0
votes
0answers
10 views

How to test measurement invariance with PLS?

I have two questions about structural equations modeling: To perform comparisons of model estimates for different groups (3 clusters), do I always have to analize measurement invariance first? If ...
0
votes
0answers
10 views

Multiple comparisons with Dunnetts t3 or Dunnetts c

I have a dataset with un-even variance and un-equal class sizes (This was a random sample of land cover classes). After using ANOVA to determine significant differences exist among classes, I ran ...
0
votes
0answers
16 views

Multiple comparison in the unequal variance case

I have found procedures such as Tamhanes T2, Dunnets T3 and the Games & Howell procedure that deal with unequal variances in the one-way model. However, I have a Randomized complete block design, ...
2
votes
1answer
37 views

Does one need to adjust for multiple comparisons when using MANOVA?

There are 10 dependent variables and 2 groups. When I wanted to compare 10 dependent variables between two groups, I performed MANOVA. I wonder if there is a multiple comparison problem in that case. ...
4
votes
1answer
49 views

Does R have post hoc tests robust to unequal sample sizes/population variances?

While reading Discovering Statistics Using R pp. 431-432, Dr. Field says that "There are a variety of tests designed to deal with these situations [multiple comparison procedures with unequal ...
0
votes
0answers
21 views

When are multiple linear regressions independent?

I'm doing many pairwise linear regressions on a set of variables. For example x as the dependent variable vs y, ...
0
votes
1answer
49 views

Hypothesis testing multiple choice question with single answer

I have a survey questions with five options and I ask respondents to pick their single top choice. What test should I use to figure out if the top voted answer is statistically more popular than the ...
0
votes
0answers
5 views

multi-item constructs/traits on a 5-point Likert scale as predictor variables

Can we use multi-item constructs/traits measured on a 5-point Likert scale as predictor variables in regression?
2
votes
0answers
22 views

Multiple ANOVAs, shall I correct the homogeneity of variance test?

I want to run nine different ANOVAs on different variables that may change among groups. The ANOVAs are preceded by the Levene test of variance homogeneity. Of course, I want to correct for multiple ...
6
votes
1answer
50 views

Order of items in a questionnaire

Does the order of the items in a questionnaire matter? If so, how do I know which item should go first, second, third etc. when devising a questionnaire? Thanks
1
vote
1answer
38 views

Multiple comparisons for correlation matrix?

Does the computation of an N x N correlation matrix for N unrelated variables require multiple comparisons correction for all the computed pairwise correlations (assuming each computed correlation is ...
4
votes
2answers
54 views

What is the name of a plot of - log 10 of p values in multiple testing?

Under the null, the p-value is uniformly distributed between 0 and 1. Taking the negative of the log base 10 of many such p-values should follow an exponential distribution. You can show quantiles of ...
0
votes
0answers
30 views

Can I use the Williams type contrast from `multcomp` package to perform a Williams test(Williams, 1971)?

Using Williams type contrast in the multiple comparison test with multcomp package will give different test statistics and test p-values for the same dataset and ...
2
votes
2answers
179 views

How to create ROC curve to assess the performance of regression models?

I knew that, ROC curve are use to assess the performance of classifiers. But is it possible to generate ROC curve for the regression model? If yes, How?
0
votes
1answer
61 views

Dummy variables in a multiple regression

I have a linear regression model with 3 independent variables (let's say A1, A2, A3) and 2 ...
0
votes
0answers
27 views

How to normalize ranked data

I am doing some machine learning and need help with the stats aspect of my problem. I have a number of addresses of webpages and some features for these webpages. I am running TF-IDF on the webpage ...
2
votes
0answers
63 views

Performance testing: detecting changes in distributions over time

I have an interesting application I’d like some advice on. A common task when working with stochastic systems in an engineering context is testing for regressions and improvements in performance. ...
1
vote
1answer
38 views

How to measure the influence of an index on the correlation between 2 other indices

I have 3 raster datasets. All 3 are values of continuous indices varying between 1 and -1. These are satellite measurements of the same area, divided in a raster. Each cell has a value for every ...
2
votes
0answers
20 views

Adjusting data for measurement errors

I have approximately 500,000 data points that were mechanically measured over about 2000 days. The equipment is manually set up each day, which likely induces some error. What I am attempting to do ...
2
votes
0answers
57 views

Randomization test and comparisons of many to 1

I have about 80 different groups, each with 3 replicates which I would like to compare against a control having always a value of 1. As I have a low number of replicates and many groups having 2 ...
1
vote
2answers
41 views

Does filtering of data by effect size violate some assumption of P value adjustment methods?

I have pre- and post-treatment continuous data for a large number of variables that I am analyzing for treatment effect. Normally I would obtain the P values and then adjust them for multiple testing ...
0
votes
0answers
37 views

Multiple Comparisons Correction on Peason's Chi-Squared Tests

I have a dataset consisting of 1000 samples (rows). Each sample consists of 300 categorical data points (columns). The process of generating these data points has a known probability of generating an ...
3
votes
0answers
56 views

Regression analysis to estimate affinity or antipathy among celebrity wedding parties

Please consider the following plot of a graph: Each node represents a wedding party while each edge is the relation between the two parties expressed in terms of the number of people who attended ...
1
vote
0answers
33 views

Adjusted p-values across hypotheses with different power?

Simple question with no simple answer. Let's say I compute 100 p-values from 100 hypotheses that have similar power (i.e. similar sample size). Now let's say I produce 1000 p-values from a series of ...
1
vote
2answers
82 views

How to compare multiple proportions between multiple groups

I measured prey capturing success (=proportion of successful captures divided by the total number of prey captured) in four fish species (species 1: n=19, species 2: n=18, species 3: n=4, species 4: ...
0
votes
0answers
50 views

How to compare several learning algorithms in a single domain

I want to compare several learning algorithms in a single domain. In the book " Evaluating Learning Algorithms" I saw the procedures to compare two learning algorithms in a single domain or multiple ...
0
votes
0answers
16 views

How to run a Simes test?

I'm looking to set up a Simes test in R. I'm trying to show that a Simes test is conservative with when a positive correlation exists. I know how correct the p-values using the p.adjust function, ...
9
votes
0answers
172 views

How much smaller can $p$ values from ANOVA's $F$-test be vs. those from multiple $t$-tests on the same data?

Intro: Having noted the attention received today by this question, "Can ANOVA be significant when none of the pairwise t-tests is?," I thought I might be able to reframe it in an interesting way that ...
1
vote
1answer
37 views

Setting significance levels for multiple testing

I have data from a number of data sets I am looking to find a correlation between measured volatiles. I have performed a t-test and have a p-value for each set of data. I now need to determine if I ...
2
votes
0answers
80 views

The correct use of 'statistical significance'

Can chi-squared tests be used for making statements like this or is this an inaccurate analysis of my data? The majority of the groups (n=9) affected by disease ranged in prevalence from 5.56% to ...
1
vote
0answers
48 views

How do I run a Tamhane's T2 multiple comparison with R? [closed]

I have taken samples from various locations and want to compare the results for the locations. The data is normal, but has unequal variance, so I cannot use an ANOVA and Tukey's HSD to perform ...
1
vote
0answers
100 views

Plot result of Nemenyi's test [closed]

Could you please tell me which R function/library (not necessarily R) to use to plot Nemenyi's test result with the critical differences? Below there is an example which was retrieved from this* ...
0
votes
0answers
24 views

Should one compute multiple q-values when power of tests differs or one global q-value?

Background: If I have 40 hypotheses that represent the same statistical test based on equally large data sets, I can compute 40 q-values (Benjamini-Hochberg). True Problem What if I have 100 sets ...
0
votes
0answers
100 views

SPSS: Crosstabs multiple comparison

I have a table with 11 columns and 2 rows. Column 1 is the control group and I want to compare all the other columns with it. I'm not interested in compraing columns 2 - 11 with each other and I'm ...
3
votes
3answers
216 views

Hypothesis test for difference in medians among more than two samples

Question The test scores of three groups of people are saved as separate vectors in R. ...
0
votes
2answers
50 views

Misunderstanding of use of standard deviation across groups

I'm trying to understand standard deviation. I understand how to determine standard deviation across a group, but I am not understanding using standard deviation to compare groups. Am I attempting to ...
1
vote
2answers
95 views

Adjusting a subset of P values to correct for multiple testing

I have a list of 300 $p$ values selected from all $p$ values obtained in multiple testing such that $p<.05$. I do not know the values of $p>.05$ or their number. The $p$ values are not corrected ...
1
vote
0answers
19 views

How do you control FWER when testing multiple nested models with likelihood ratio tests?

Suppose I have three different models $M_1, M_2$ and $M_3$, these models are nested such that $M_1 \subseteq M_2 \subseteq M_3$. I perform two likelihood ratio tests, $M_1$ against $M_3$ and $M_2$ ...
0
votes
3answers
91 views

When to use “difference of mean change” versus “difference of final values”

Question: I want to test a medicine and I have two groups of people with baseline blood pressure, one I give medicine A and the other one medicine B. After 6 months, I measure their blood pressure. ...
1
vote
0answers
30 views

How can I evaluate multiple hypotheses simultanousely using the bootstrap?

Suppose you have two null hypothesis for two parameters, $\theta_1$ and $\theta_2$ based on the same variable $X$. So for example, $\theta_1$ is mean of $X$ in group 1, whereas $\theta_2$ is mean of ...
0
votes
2answers
75 views

Testing which groups have had a significant change

I have three groups with two hundred individuals each, and each individual within a group has a pre and post treatment measurement. A single group contains individuals with similar pre measurement ...
0
votes
1answer
61 views

How can I compare variables with uneven data sets?

I'm doing a stationary sport in which I use different colored equipment for different weather. I am recording my scores along with the color of the equipment and the weather in which I am using it. I ...
3
votes
0answers
55 views

Is Bonferroni correction too liberal for some dependent hypotheses?

I frequently read that Bonferroni correction also works for dependent hypotheses. However, I don't think that is true and I have a counter example. Can somebody please tell me (a) where my mistake is ...