Questions tagged [p-value]

In frequentist hypothesis testing, the $p$-value is the probability of a result as extreme (or more) than the observed result, under the assumption that the null hypothesis is true.

863 questions with no upvoted or accepted answers
Filter by
Sorted by
Tagged with
2k views

Are these equivalent (for p-values): super-uniform, stochastically larger than / dominating the uniform, conservative?

In the literature and online, I have found three different wordings that I think refer to the same concept: stochastically larger than uniform (which I take is ...
• 231
19k views

How to find p value using estimate and standard error?

I am trying to check statement on p. 23 of Data Analysis Using Regression and Multilevel/Hierarchical Models For example, consider two independent studies with effect estimates and standard errors ...
• 131
214 views

Efficient nonparametric estimation of confidence intervals and p-values for nonlinear regression

I'm estimating parameters for a complex, "implicit" nonlinear model $f(\mathbf{x}, \boldsymbol{\theta})$. It's "implicit" in the sense that I don't have an explicit formula for $f$: its value is the ...
• 18.3k
675 views

Skew of p-value distribution under composite null hypotheses

On this page it says ...if HA holds, the p-values have a distribution for which values near 0 are more likely than values near 1. However the p-values may have a distribution that is not ...
• 12.7k
7k views

How to analyze and interpret A/B test results via Bootstrap method?

We've run a split test of a new product feature and want to measure if the uplift on revenue is significant. Our observations are definitely not normally distributed (most of our users don't spend, ...
• 441
75 views

Has the reproducibility crisis affected confidence intervals as well?

The reproducibility crisis has given many pause over the value (?) of $p$-values to measure the relevance of statistical findings. Given the interpretation of a $p$-value and some knowledge of ...
• 63.7k
396 views

Is this popular approach to Bootstrap hypothesis testing correct?

A widely cited paper "Statistical Significance Tests for Machine Translation Evaluation" proposes usage of Bootstrap for evaluating significance of difference across systems for Machine Translation (...
• 1,288
50 views

Implications of Different Bootstrap Procedures for Estimating Difference in Means

I'm having trouble understanding the difference between two bootstrap procedures to evaluate the difference in means between two samples. As an example, consider the following scenario: My goal is to ...
250 views

Comparing multiple frequency distributions from data and from simulation

I am attempting to analyze the shuffling in the card based online game Magic Arena, which I suspect has a specific bug that causes the distribution of outputs to be biased. Is the approach I have ...
• 161
193 views

Non-uniform p-values from hoeffd function in Hmisc when data sets are independent

When using the function hoeffd in the CRAN package Hmisc I get unusual p-values for pairs of data sets that are independent. The function hoeffd is an implementation of Hoeffding's $D$ statistic. ...
• 722
4k views

How do I get the p value of AD test using the results of scipy.stats.anderson()

I am conducting an Anderson Darling test for normality using scipy.stats.anderson() command in python. I am getting test statistic, critical values at various ...
5k views

I know the p-value and I may know what FDR (false discovery rate) do and its goal. But I confuse between q-value (often known as FDR) and adjusted p-value (p.ajust ...
53 views

How does Lindley compare a Bayes factor and a p-value?

I was reading this paper by Dennis Lindley ("Analysis of a Wine Tasting", J. Wine Econ. 2006). Statistically, the paper is a straightforward analysis of a $10\times 11$ two-way table. To test whether ...
• 2,096
2k views

Use Chi-Squared or Binomial Test if Distribution is not Known?

Suppose you have a set of data (eg. [a, b, a, a, b, b, etc.]), and you have the suspicion that the set of data follows the binomial distribution. Your Null Hypothesis is: The probability of success ...
• 51
86 views

Analysis of "The advantage of short paper titles"

After seeing the paper, The advantage of short paper titles showing a relationship between paper length rank and citation rank, I wondering what to make of the results. As a layman, my questions are: ...
• 8,948
816 views

Using the hypergeometric distribution for skipping events in transcriptome sequencing

My question is inspired by this post. However, its a bit more complicated than that to explain. I hope I succeed. I work with RNA-Seq data on alternative splicing in plants. For this discussion, lets ...
• 854
1k views

How to estimate False Discovery Rate from p-value distribution?

I have learned many models and I calculated p-values for the cross-validation errors. I want to select significant models based on the false discovery rate (FDR). How can I estimate the FDR from p-...
• 2,743
72 views

P value and sample size

Say for a 2x2 experiment, the power analysis said the sample size should be over 1000 to detect 0.1 effect size. Right now, with the sample size of below 100, the effect size of .12 is insignificant ...
212 views

Calculating confidence intervals for coefficients in ordinal logistic regression

I've fitted an ordinal logistic regression model in R using MASS::polr and I am looking to compute confidence intervals for the fitted coefficients. Here is the (...
219 views

• 51
75 views

What can a p-value (& sign) tell me about the marginal posterior distribution of a model parameter, and when?

EDIT: The tl;dr here would broadly be: given that both frequentist standard errors and a quadratic approximation of a Bayesian joint posterior can be obtained from the square root of the diagonal ...
371 views

How to calculate p-value to compare AUPRC for two models?

I've built two deep learning models and was looking for a way to compare the two models in terms of the area under the precision recall curve terms. I know that the DeLong method is recommended for ...
• 55
$p$-Values: Standardized coefficients vs. partial $R^2$ vs. $\Delta R^2$
In this excellent answer, the definitions of, and differences between, the three quantities in the title of this question are laid out. My question concerns the relationship between their $p$-values. ...