Questions tagged [hypothesis-testing]

Hypothesis testing assesses whether data are inconsistent with a given hypothesis rather than being an effect of random fluctuations.

3,570 questions with no upvoted or accepted answers
Filter by
Sorted by
Tagged with
14 votes
0 answers
1k views

Testing for a significant difference between ML estimates: Likelihood ratio or Wald test?

I am trying to test whether or not there is a significant difference between maximum likelihood estimates of two genetic parameters (selection and dominance) across two environments with genotype data ...
user avatar
  • 141
12 votes
0 answers
1k views

Understanding Sequential Probability Ratio Test (SPRT) Likelihood Ratio

I am a software developer looking to develop an alternative for the simple hypothesis testing scheme described here. In short, the test works as follows: Two URLs are compared for their ability to ...
user avatar
  • 653
10 votes
1 answer
204 views

Unit testing sampling methods

I'm writing a bit of code that makes pretty heavy use of sampling (eg, MCMC, Particle Filters, etc), and I would really like to test it to make sure that it's doing what I think it is before claiming ...
user avatar
9 votes
0 answers
434 views

How is it possible to control false discovery rate (FDR) without knowing the power and the prevalence of the nulls?

If we have p-value 0.05, to calculate probability of our discovery to be false positive, we need to use complex formula with prevalence (prior probability) and statistical power. If we have lots of ...
user avatar
9 votes
1 answer
2k views

How to test for the difference in skewness of two samples?

I have two samples. From looking at their densities, one appears symmetrical and the other from some right-tailed distribution. I would like to test that the two do not have the same skewness (...
user avatar
9 votes
1 answer
1k views

How to handle multiple measurements per participant, with categorical data?

I've done an experiment where I've collected measurements from a number of participants. Each relevant data point has two variables, both categorical: in fact, each variable has two possible values (...
user avatar
  • 5,953
8 votes
0 answers
661 views

Are these equivalent (for p-values): super-uniform, stochastically larger than / dominating the uniform, conservative?

In the literature and online, I have found three different wordings that I think refer to the same concept: stochastically larger than uniform (which I take is ...
user avatar
8 votes
0 answers
3k views

Assumptions of correlation test vs regression slope test (significance testing)

If my understanding is correct, then the test on a regression slope in a simple bivariate regression - i.e. the test of $\mathcal{H}_0$: $b = 0$ in $Y' = a + bX$ and the test of a correlation, i.e. $...
user avatar
  • 331
8 votes
0 answers
1k views

Proof of Kolmogorov-Smirnov test

Could someone provide me a reference, preferably a book, where I can find detailed proofs and explanations of the Kolmogorov-Smirnov test (including the two-sample variant) and the derivation of the $...
user avatar
  • 221
8 votes
1 answer
5k views

How to analyze and interpret A/B test results via Bootstrap method?

We've run a split test of a new product feature and want to measure if the uplift on revenue is significant. Our observations are definitely not normally distributed (most of our users don't spend, ...
user avatar
  • 401
8 votes
1 answer
4k views

Comparing two F-test statistics

I have two groups A and B, each of which consists of 5 samples. Each sample is described in a vector of length (>1000) of continuous numeric values (characteristics). I want to test if the sample in ...
user avatar
  • 485
7 votes
0 answers
248 views

The Mediterranean diet - statistics issues with the updated paper

In 2013, a group of researchers published a paper with results on a randomized trial of the Mediterranean diet, finding that it appears to have significant health benefits. Today, they retracted ...
user avatar
  • 5,953
7 votes
0 answers
1k views

Combining Z-scores using Stouffer's approach

Quick background: I require to obtain a single 'score' for comparison between experimental and simulated data. The parameter being tested is the height of various parts (num = 5) of the model. For the ...
user avatar
7 votes
1 answer
2k views

2-Sample Proportions z-test vs Fisher's Exact Test

I want to evaluate the difference in proportions in declined applications and exceptions from policy from one quarter of data to the next. My sample sizes are generally not small (several hundred to ...
user avatar
  • 71
7 votes
0 answers
380 views

Is this popular approach to Bootstrap hypothesis testing correct?

A widely cited paper "Statistical Significance Tests for Machine Translation Evaluation" proposes usage of Bootstrap for evaluating significance of difference across systems for Machine Translation (...
user avatar
  • 1,168
7 votes
0 answers
628 views

Skew of p-value distribution under composite null hypotheses

On this page it says ...if HA holds, the p-values have a distribution for which values near 0 are more likely than values near 1. However the p-values may have a distribution that is not ...
user avatar
  • 11.5k
7 votes
0 answers
162 views

Hypothesis test on the Euclidean length of an unknown vector

Question Suppose I observe a vector $\mathbf{x}=[X_1 \ldots X_n]$, where each $X_i=m_i+n_i$, with $n_i$ being an independent zero-mean Gaussian random variable with variance $\sigma^2$ (i.e. $n_i\sim\...
user avatar
  • 1,059
7 votes
0 answers
143 views

Test for difference of distributions on a torus

I have two circular dependent variables and would like to test for a difference in the distributions (presumably circular means) between multiple treatment groups. There are a number of multivariate ...
user avatar
  • 447
6 votes
0 answers
168 views

Why use the student's t-test rather than z-score?

Suppose we are given IID r.v's $X_1, \ldots, X_n$ that are not necessarily normally distributed. Mean $\mu$ and standard deviation $\sigma$ are unknown and we want to construct a confidence interval ...
user avatar
6 votes
0 answers
639 views

A specific example of two-sided chi-squared test

First of all I have read the answers to this question, but I'm not happy with them, I feel that they miss the point that I'm willing to address here. I'm looking at a chi-squared test for ...
user avatar
6 votes
1 answer
142 views

Correlation between logrank (log-rank) test statistics with common control

Say I have a $K$ arm experiment that generates survival (time-to-event) endpoints. There are $K-1$ experimental arms and a single control arm. Say I compute a log rank test statistic comparing the ...
user avatar
  • 2,012
6 votes
0 answers
218 views

Comparing multiple frequency distributions from data and from simulation

I am attempting to analyze the shuffling in the card based online game Magic Arena, which I suspect has a specific bug that causes the distribution of outputs to be biased. Is the approach I have ...
user avatar
  • 161
6 votes
0 answers
3k views

Two-Sample Hypothesis Test for Difference Between Two Proportions: Pooled vs Unpooled? How to Choose?

When are you supposed to use the pooled formula vs the unpooled formula? My understanding is that you use the pooled formula when testing a hypothesis where the null hypothesis is $H_0 = 0$ and the ...
user avatar
  • 285
6 votes
3 answers
783 views

Significance test for entropy?

Is there any way to test the difference of entropy given frequency table? For example, let's say we have dice 1 and dice 2, and we experimented with them and they showed up like ...
user avatar
  • 1,121
5 votes
0 answers
327 views

Pooled variance for post hoc tests for contingency table $\chi^{2}$ tests

The 2-by-2 contingency table test provides an alternate hypothesis testing approach to the $z$ test for proportion difference. Conveniently enough, a 2-by-k contingency table test provides an omnibus ...
user avatar
  • 26.4k
5 votes
1 answer
2k views

T-test for regression coefficients obtained from Ridge, LASSO etc

In ordinary least squares, for example in an experimental design case, I obtain the regression coefficents by: $ \hat B = {({X^t}{X})}^{-1}X^ty$ Then, my null hypothesis for each coefficent is: $...
user avatar
  • 1,392
5 votes
0 answers
164 views

Exact Null Distribution with Ties

I am interested in deriving exact null distributions for small-sample test statistics with non-trivial ties. Not fundamentally continuous variables that happen to have a few repeated values, but ...
user avatar
  • 2,181
5 votes
1 answer
114 views

Statistical test to see whether we drew overly-balanced samples

I have an urn with $R$ red and $B$ black balls. $R$ and $B$ are large numbers. I draw some $n$ samples of different sizes without replacement until the urn is empty. The sizes of the samples are not a ...
user avatar
5 votes
0 answers
664 views

Hypothesis testing for Gamma distribution

I have a sample $X_1,...,X_n \sim \Gamma(\alpha, \beta)$, where $\alpha, \beta$ - unknown parameters of Gamma distribution. How to build a test for testing $H_0:\alpha=1$ against $H_1:\alpha>1$?
user avatar
  • 51
5 votes
0 answers
378 views

Cross model comparison of quantile regression coefficients

I am looking for a way to compare coefficients obtained from quantile regression. The two surveyed models are nested, estimated on the same sample and for the same quantile. $$ Y = \beta_1X+\epsilon_2\...
user avatar
  • 51
5 votes
0 answers
2k views

Use Chi-Squared or Binomial Test if Distribution is not Known?

Suppose you have a set of data (eg. [a, b, a, a, b, b, etc.]), and you have the suspicion that the set of data follows the binomial distribution. Your Null Hypothesis is: The probability of success ...
user avatar
  • 51
5 votes
0 answers
453 views

Hypothesis testing for binomial distribution: a simple real-life case

I'm refreshing (or maybe just acquiring) some stats skills, and I have a real-life situation, probably very simple since it's quite close to a typical example from stats courses, that I would like to ...
user avatar
  • 51
5 votes
1 answer
935 views

Multivariate differences between groups controlling for one factor (MANOVA)

I have a sample of 100 participants who have scores on 5 different variables (V1-V5). Some participants took part in a workshop, others did not. I am interested in investigating the influence of ...
user avatar
  • 51
5 votes
0 answers
6k views

Zero-inflated model: non-finite value supplied by optim

So I have the following model predicting the presence of an animal on a certain spot. As a time unit quarter is initially used, but for one of the species of animals there is some (little) interesting ...
user avatar
  • 271
5 votes
0 answers
471 views

What model contains Wilcoxon's rank sum test?

As Frank Harrel mentions in the answer to Logistic regression and Wilcoxon test, Wilcoxon's rank sum test (aka Mann-Whithey test) is contained in the proportional odds logistic regression, which would ...
user avatar
  • 10.6k
5 votes
0 answers
3k views

Testing for conditional independence: What's the correct way?

My goal is to check if two variables $X$ and $Y$ are conditionally independent given $Z$. For simplicity, let's assume the joint distribution is multivariate normal. In this case, we can compute ...
user avatar
  • 1,047
5 votes
1 answer
101 views

Testing a proportion in an online setting

I work in an online security setting. My goal is to detect if the number of locked accounts per time unit is stable or not. I've tried several approaches, detailed below, but I am not satisfied yet. ...
user avatar
  • 2,032
5 votes
0 answers
1k views

Hypothesis testing with non-iid observations

I have a dataset with observations which violate the independent and identically distributed (iid) assumption. The dataset is composed by a number of post-click conversion events (purchases on a ...
user avatar
  • 293
5 votes
0 answers
255 views

Asymmetry of the Kullback-Leibler distance in hypothesis testing

My question is related to the asymmetry of the Kullback-Leibler distance. I'm using the discrete definition of the Kullback-Leibler distance, so we have: $$ KL(p,q) = \sum_{s \in S} p(s) \log\left( \...
user avatar
  • 51
5 votes
0 answers
703 views

appropriate hypothesis test for non-negative quantity

This question is intentionally written in a very abstract way, because I am interested in a generic solution, or for pointers to different possible concrete cases in different fields: I am measuring ...
user avatar
  • 2,859
5 votes
0 answers
410 views

How Should I Find Statistically Significant Differences Between Two Markov Models?

Suppose I have N Markov Models of M states representing the behaviour patterns of 2 different groups (note: fully observable models, no hidden states), and have stored each model as a matrix of ...
user avatar
5 votes
0 answers
2k views

Neyman-Pearson lemma: critical region and hypothesis testing

Let $X_1,X_2,...,X_n$ be i.i.d r.v's with common p.d.f. $$ \mbox f(x)=\frac{x^5e^{-x/\theta}}{5!\theta^6} $$ where $\theta$ > 0. Show that the Neyman-Pearson lemma produces a test of $H_0: \...
user avatar
5 votes
0 answers
3k views

Ljung–Box test for a multivariate time series?

From Tsay's Analysis of Financial Time Series, For a univariate weakly stationary time series $r_t$, its sample autocorrelation function $\hat{\rho}_l$ is defined as: and the Ljung-Box test is ...
user avatar
  • 17.9k
5 votes
0 answers
4k views

One-Sample t Test and Likelihood Ratio Test

Concerning testing the hypotheses about the mean $\mu$ of a single population I know of things like the One-Sample $t$ test where there is a null hypothesis: $H_0: \mu = \mu_0$, and a test statistic $...
user avatar
5 votes
0 answers
751 views

Confirming the detailed calcs for NPS Margin of error

Based on this very good post How can I calculate margin of error in a NPS (Net Promoter Score) result? I've pulled together the detailed calculations to perform the test for the generalised NPS ...
user avatar
5 votes
0 answers
1k views

Binomial mid-p value

I've been under the impression that the mid-$p$ values generally control the Type I error, and consequently confidence intervals based on mid-$p$ values control the coverage. However I have checked ...
user avatar
5 votes
0 answers
788 views

Using the hypergeometric distribution for skipping events in transcriptome sequencing

My question is inspired by this post. However, its a bit more complicated than that to explain. I hope I succeed. I work with RNA-Seq data on alternative splicing in plants. For this discussion, lets ...
user avatar
  • 834
4 votes
0 answers
71 views

How does R's aov function 'work better' with orthogonal contrasts?

The help page for R's aov function states: The default ‘contrasts’ in R are not orthogonal contrasts, and aov and its helper functions will work better with such ...
user avatar
  • 1,003
4 votes
1 answer
179 views

Testing equality of quantile regression slopes at different quantiles

How do I test if the quantile regression slopes are equal for different quantiles? E.g. I run a quantile regression at 5% quantile, 50% quantile (median) and 95% quantile and obtain the slope ...
user avatar
4 votes
0 answers
133 views

P-value of LR test

I've been studying more about GLRT (Generalized Likelihood Ratio Tests) and I came up with the following problem. Let $X\sim N(\theta,1)$ and consider the hypothesis $H_0:\theta\in[a,b]$ against $H_1:\...
user avatar

1
2 3 4 5
72