Questions tagged [hypothesis-testing]

Hypothesis testing assesses whether data are inconsistent with a given hypothesis rather than being an effect of random fluctuations.

3,192 questions with no upvoted or accepted answers
Filter by
Sorted by
Tagged with
12
votes
0answers
1k views

Testing for a significant difference between ML estimates: Likelihood ratio or Wald test?

I am trying to test whether or not there is a significant difference between maximum likelihood estimates of two genetic parameters (selection and dominance) across two environments with genotype data ...
10
votes
0answers
919 views

Understanding Sequential Probability Ratio Test (SPRT) Likelihood Ratio

I am a software developer looking to develop an alternative for the simple hypothesis testing scheme described here. In short, the test works as follows: Two URLs are compared for their ability to ...
9
votes
1answer
2k views

How to test for the difference in skewness of two samples?

I have two samples. From looking at their densities, one appears symmetrical and the other from some right-tailed distribution. I would like to test that the two do not have the same skewness (...
8
votes
1answer
1k views

How to handle multiple measurements per participant, with categorical data?

I've done an experiment where I've collected measurements from a number of participants. Each relevant data point has two variables, both categorical: in fact, each variable has two possible values (...
8
votes
1answer
157 views

Unit testing sampling methods

I'm writing a bit of code that makes pretty heavy use of sampling (eg, MCMC, Particle Filters, etc), and I would really like to test it to make sure that it's doing what I think it is before claiming ...
7
votes
0answers
336 views

Are these equivalent (for p-values): super-uniform, stochastically larger than / dominating the uniform, conservative?

In the literature and online, I have found three different wordings that I think refer to the same concept: stochastically larger than uniform (which I take is ...
7
votes
0answers
87 views

How is it possible to control false discovery rate (FDR) without knowing the power and the prevalence of the nulls?

If we have p-value 0.05, to calculate probability of our discovery to be false positive, we need to use complex formula with prevalence (prior probability) and statistical power. If we have lots of ...
7
votes
0answers
202 views

Is the Berry-Esseen theorem useful for justifying normality?

The Kolmogorov-Smirnov (KS) test tells one how confident they can be that a sample comes from a hypothesized distribution. It is my understanding that this test can be used to justify whether or not ...
7
votes
0answers
978 views

Proof of Kolmogorov-Smirnov test

Could someone provide me a reference, preferably a book, where I can find detailed proofs and explanations of the Kolmogorov-Smirnov test (including the two-sample variant) and the derivation of the $...
6
votes
1answer
2k views

How is pairwise PERMANOVA/adonis a valid non-parametric approach for pairwise comparisons

Assume that we have taken independent random samples of several individuals from 5 locations that represent 5 populations. The design is fairly unbalanced: the number of individuals sampled from each ...
6
votes
0answers
7k views

Assessing A/B test results using Python

Problem's setting Assume having results of an A/B test. You let you users experience two variants of your website and you counted how many converted: ...
6
votes
0answers
925 views

Combining Z-scores using Stouffer's approach

Quick background: I require to obtain a single 'score' for comparison between experimental and simulated data. The parameter being tested is the height of various parts (num = 5) of the model. For the ...
6
votes
1answer
1k views

2-Sample Proportions z-test vs Fisher's Exact Test

I want to evaluate the difference in proportions in declined applications and exceptions from policy from one quarter of data to the next. My sample sizes are generally not small (several hundred to ...
6
votes
0answers
771 views

Testing for statistical differences of quantile regression line slopes

If I were to compare the statistical similarity between the slopes of OLS regression lines from two independent samples, I would use a t-test to test if the slopes are equal or not. I'd like to ...
6
votes
0answers
604 views

Skew of p-value distribution under composite null hypotheses

On this page it says ...if HA holds, the p-values have a distribution for which values near 0 are more likely than values near 1. However the p-values may have a distribution that is not ...
6
votes
0answers
136 views

Test for difference of distributions on a torus

I have two circular dependent variables and would like to test for a difference in the distributions (presumably circular means) between multiple treatment groups. There are a number of multivariate ...
6
votes
1answer
946 views

How do I compare date-ranges from a time series?

I have a time series which contains monthly readings for air pollution in a city. The seasonality from this time series has been removed. Given two date ranges, for example Jan-Aug 2008 and Jan-Aug ...
5
votes
1answer
147 views

Two sample test for exponential distribution with only two observations

Suppose we have two independent random variables $X_1 \sim \exp(\lambda_1)$ and $X_2 \sim \exp(\lambda_2)$ . Now, we are given just one observation each from the two distributions above, say $S_1$ and ...
5
votes
1answer
52 views

When to test For Equality of Medians, and when Stochastic Equality?

In statistics, we are very often interested in investigating whether some score (let's say life satisfaction) tends to be bigger in one population (let's say rich people) compared to another ...
5
votes
1answer
88 views

Correlation between logrank (log-rank) test statistics with common control

Say I have a $K$ arm experiment that generates survival (time-to-event) endpoints. There are $K-1$ experimental arms and a single control arm. Say I compute a log rank test statistic comparing the ...
5
votes
0answers
160 views

Comparing multiple frequency distributions from data and from simulation

I am attempting to analyze the shuffling in the card based online game Magic Arena, which I suspect has a specific bug that causes the distribution of outputs to be biased. Is the approach I have ...
5
votes
0answers
239 views

The Mediterranean diet - statistics issues with the updated paper

In 2013, a group of researchers published a paper with results on a randomized trial of the Mediterranean diet, finding that it appears to have significant health benefits. Today, they retracted ...
5
votes
0answers
263 views

Pooled variance for post hoc tests for contingency table $\chi^{2}$ tests

The 2-by-2 contingency table test provides an alternate hypothesis testing approach to the $z$ test for proportion difference. Conveniently enough, a 2-by-k contingency table test provides an omnibus ...
5
votes
1answer
2k views

T-test for regression coefficients obtained from Ridge, LASSO etc

In ordinary least squares, for example in an experimental design case, I obtain the regression coefficents by: $ \hat B = {({X^t}{X})}^{-1}X^ty$ Then, my null hypothesis for each coefficent is: $...
5
votes
0answers
151 views

Exact Null Distribution with Ties

I am interested in deriving exact null distributions for small-sample test statistics with non-trivial ties. Not fundamentally continuous variables that happen to have a few repeated values, but ...
5
votes
0answers
55 views

Statistical test to see whether we drew overly-balanced samples

I have an urn with $R$ red and $B$ black balls. $R$ and $B$ are large numbers. I draw some $n$ samples of different sizes without replacement until the urn is empty. The sizes of the samples are not a ...
5
votes
0answers
602 views

Hypothesis testing for Gamma distribution

I have a sample $X_1,...,X_n \sim \Gamma(\alpha, \beta)$, where $\alpha, \beta$ - unknown parameters of Gamma distribution. How to build a test for testing $H_0:\alpha=1$ against $H_1:\alpha>1$?
5
votes
0answers
346 views

Cross model comparison of quantile regression coefficients

I am looking for a way to compare coefficients obtained from quantile regression. The two surveyed models are nested, estimated on the same sample and for the same quantile. $$ Y = \beta_1X+\epsilon_2\...
5
votes
0answers
1k views

Use Chi-Squared or Binomial Test if Distribution is not Known?

Suppose you have a set of data (eg. [a, b, a, a, b, b, etc.]), and you have the suspicion that the set of data follows the binomial distribution. Your Null Hypothesis is: The probability of success ...
5
votes
0answers
438 views

Hypothesis testing for binomial distribution: a simple real-life case

I'm refreshing (or maybe just acquiring) some stats skills, and I have a real-life situation, probably very simple since it's quite close to a typical example from stats courses, that I would like to ...
5
votes
1answer
869 views

Multivariate differences between groups controlling for one factor (MANOVA)

I have a sample of 100 participants who have scores on 5 different variables (V1-V5). Some participants took part in a workshop, others did not. I am interested in investigating the influence of ...
5
votes
0answers
5k views

Zero-inflated model: non-finite value supplied by optim

So I have the following model predicting the presence of an animal on a certain spot. As a time unit quarter is initially used, but for one of the species of animals there is some (little) interesting ...
5
votes
0answers
406 views

What model contains Wilcoxon's rank sum test?

As Frank Harrel mentions in the answer to Logistic regression and Wilcoxon test, Wilcoxon's rank sum test (aka Mann-Whithey test) is contained in the proportional odds logistic regression, which would ...
5
votes
0answers
3k views

Assumptions of correlation test vs regression slope test (significance testing)

If my understanding is correct, then the test on a regression slope in a simple bivariate regression - i.e. the test of $\mathcal{H}_0$: $b = 0$ in $Y' = a + bX$ and the test of a correlation, i.e. $...
5
votes
0answers
686 views

Confidence-interval / p-value duality vs. frequentist interpretation of CIs

Many sources suggest that there is a duality between confidence intervals and hypothesis testing.(*) But I'm having trouble making sense of this philosophically. The frequentist interpretation of a ...
5
votes
0answers
345 views

Is this popular approach to Bootstrap hypothesis testing correct?

A widely cited paper "Statistical Significance Tests for Machine Translation Evaluation" proposes usage of Bootstrap for evaluating significance of difference across systems for Machine Translation (...
5
votes
0answers
3k views

Testing for conditional independence: What's the correct way?

My goal is to check if two variables $X$ and $Y$ are conditionally independent given $Z$. For simplicity, let's assume the joint distribution is multivariate normal. In this case, we can compute ...
5
votes
1answer
94 views

Testing a proportion in an online setting

I work in an online security setting. My goal is to detect if the number of locked accounts per time unit is stable or not. I've tried several approaches, detailed below, but I am not satisfied yet. ...
5
votes
0answers
831 views

Hypothesis testing with non-iid observations

I have a dataset with observations which violate the independent and identically distributed (iid) assumption. The dataset is composed by a number of post-click conversion events (purchases on a ...
5
votes
0answers
248 views

Asymmetry of the Kullback-Leibler distance in hypothesis testing

My question is related to the asymmetry of the Kullback-Leibler distance. I'm using the discrete definition of the Kullback-Leibler distance, so we have: $$ KL(p,q) = \sum_{s \in S} p(s) \log\left( \...
5
votes
0answers
611 views

appropriate hypothesis test for non-negative quantity

This question is intentionally written in a very abstract way, because I am interested in a generic solution, or for pointers to different possible concrete cases in different fields: I am measuring ...
5
votes
0answers
405 views

How Should I Find Statistically Significant Differences Between Two Markov Models?

Suppose I have N Markov Models of M states representing the behaviour patterns of 2 different groups (note: fully observable models, no hidden states), and have stored each model as a matrix of ...
5
votes
0answers
2k views

Ljung–Box test for a multivariate time series?

From Tsay's Analysis of Financial Time Series, For a univariate weakly stationary time series $r_t$, its sample autocorrelation function $\hat{\rho}_l$ is defined as: and the Ljung-Box test is ...
5
votes
0answers
4k views

One-Sample t Test and Likelihood Ratio Test

Concerning testing the hypotheses about the mean $\mu$ of a single population I know of things like the One-Sample $t$ test where there is a null hypothesis: $H_0: \mu = \mu_0$, and a test statistic $...
5
votes
1answer
4k views

How to analyze and interpret A/B test results via Bootstrap method?

We've run a split test of a new product feature and want to measure if the uplift on revenue is significant. Our observations are definitely not normally distributed (most of our users don't spend, ...
5
votes
0answers
152 views

Hypothesis test on the Euclidean length of an unknown vector

Question Suppose I observe a vector $\mathbf{x}=[X_1 \ldots X_n]$, where each $X_i=m_i+n_i$, with $n_i$ being an independent zero-mean Gaussian random variable with variance $\sigma^2$ (i.e. $n_i\sim\...
5
votes
0answers
1k views

Binomial mid-p value

I've been under the impression that the mid-$p$ values generally control the Type I error, and consequently confidence intervals based on mid-$p$ values control the coverage. However I have checked ...
5
votes
0answers
775 views

Using the hypergeometric distribution for skipping events in transcriptome sequencing

My question is inspired by this post. However, its a bit more complicated than that to explain. I hope I succeed. I work with RNA-Seq data on alternative splicing in plants. For this discussion, lets ...
4
votes
0answers
44 views

Should I plan my AB testing accordingly to a minimum number of days or to the number of datapoints?

Here's the thing, I was planning to test a new strategy on my website. A typical AB test for a new homepage. One strategy is the current, the control; the other is the variant, the new strategy. I ...
4
votes
0answers
69 views

Finding the probability density function of Hotelling's T-squared distribution

The following image is seen on wikipedia when searching for Hotelling's T-squared distribution This is apparently the pdf of the Hotelling T-squared distribution at different parameters. However, I ...

1
2 3 4 5
64