# Questions tagged [p-value]

In frequentist hypothesis testing, the $p$-value is the probability of a result as extreme (or more) than the observed result, under the assumption that the null hypothesis is true.

2,939 questions
Filter by
Sorted by
Tagged with
1 vote
218 views

### P-Values based on Experiment Data Aggregation

I'm trying to understand which interpretation of p-values for a linear regression model is correct given different levels of aggregation. Let's say I am running a pricing experiment in a store-chain ...
1 vote
18 views

48k views

### How to calculate p-value for multivariate linear regression

Software packages that calculate regressions sometimes also return p-values. I want to understand how to calculate this p-value by hand. Here's what I think I understand: I want to calculate the ...
10 views

### Compute p-vale from beta, standard error, sample size and number of regression parameters

How do I compute the p-value if I only know the beta (i.e., the regression coefficient from a linear regression), the standard error, the sample size, and the number of regression parameters? And, is ...
39 views

### Use of multiple comparison correction and composite DV

I am investigating the effect of a single X on a composite DV, Y. I want to also investigate the effect of X on the 36 sub-components of Y. Thus, I would have to run 37 multiple regression models, ...
19 views

### Filter one-sided tests before FDR

In this blog post, the author shows one example of "weird" p-value histogram ("Scenario C"): One provided explanation is that a one-sided test was run, and the tests where the ...
29 views

### Can I use a simpler model (small R²) to get more statistically relevant results?

I am new to this kind of stuff. I am currently writing a bachelor thesis that uses data from the European Social Survey (round 7). 4 important questions that I'm using were not answered by every ...
57 views

### getting insignificant p-values but large effect size what should I do?

I used Qualtrics to conduct a survey and received 700+ responses. I also collected demographic data to disaggregate the data and observe any differences between groups. A couple questions used display ...
22 views

### Computing p-value using bootstrap bias-corrected and accelerated (BCa) confidence intervals - R

The boot.pval package throws an error when computing BCa p values. This suggested fix seems to work but I am unsure if this correct. The fix was suggested more than two years ago without having being ...
985 views

### Ranked t-test or Mann-Whitney U test for commerical A/B testing

Please help me understand how this blog post relates to this experiment. Here I try to reproduce quite old paper from Zimmerman about Mann-Whitney U test and ranked t-test (and also plain t-test) ...
7k views

### Can p-values for Pearson's correlation test be computed just from correlation coefficient and sample size?

Background: I read one article where authors report Pearson correlation 0.754 from sample size 878. Resulting p-value for correlation test is "two star" significant (i.e. p < 0.01). However, I ...
430 views

### Related low p-values that do not meet statistically significant thresholds

If I were to have multiple related measures, each with "low" p-values that do not quite meet the threshold of significance, could the fact that multiple measures produce a general trend with ...
15 views

### Significance of the coefficients of a multinomial mixed regression built with npmlt function

I fitted (using npmlt function) a mixte multinomial regression model which ran without errors. I expected that the outputs of the model include p-values that I didn't find. My need now is that someone ...
364 views

### Testing the Difference between Two Time Series ~ Similar to paired t-test

I hope everyone is doing well. I have two time series data sets (Time Series 1) and (Time Series 2). These two series are produced monthly (from 1-2020 to 12-2020) 12 data points. Thus, I took the ...
1 vote
50 views

### Test whether my new method's type I error rate correctly matches chosen alpha

I am testing two different methods for hypothesis testing, bootstrapped likelihood ratio tests, and likelihood ratio tests assuming an asymptotic distribution. I would like to see if the number of ...
158 views

### Are there alternative statistics to a p-value in NHST?

Sometimes, when the logic of p-value is explained and motivated, it happens in the following way: The concept of a random sample is introduced, together with the need to consider the possibility that ...
4k views

### How can I choose between homoscedastic and heteroscedastic?

I want to calculate the p-value between subgroups of my samples. For that, I am using the T.TEST function of Excel. But I do not understand the last parameter, type: Paired Two-sample equal variance (...
743 views

### Division of beta coefficient by p-value

Let's say I have results of a measurement for a large number of treatments (p-value, beta, t-stat) on samples from a large number of different populations. I want to compare how different the effect ...
35 views

### Determining the p-value of a test statistic, which is not distributed according to a commonly known distribution under the null hypothesis

Currently I am working in R on a project that aims to identify Dragon King events (massive outliers) in large datasets. These outliers appear for example in the city sizes in England, where London is ...
256 views

### Statistical approaches to detect overfitting in simple models

I read here that there are statistical approaches to assess whether a tractable machine learning model (e.g., a linear regression model) overfits a dataset: Simpler models that have originated in ...
1 vote
37 views

### Why can we use RMSE as a proxy for the effect size of a VAR or Granger Causality?

My question here follows a previous question by me here: Is there a meaningful option to compute something like an effect size for Granger Causality?. In this previous question I asked for options to ...
55 views

### Testing for difference in trends of binary-parameter proportion/fraction versus continuous parameter between two (or more) different populations

I have three samples of objects: sample X (total = 138), sample Y (same basic class as X, but located in a different environment; total = 186), and sample Z (total = 996). For each object, I have a ...
23 views

### The problem of double adjustment of p-values in multilevel models with categorical predictors

I have a set of around 30 multilevel models which have the same predictor variables (and the same cluster variable), but different outcome variables. Since there are so many models, I want to perform ...
21 views

I want to determine the correlation and p-value between one variable and all of the others in an R data frame, so I created a correlation matrix with cor_mat, and ...
1 vote
1k views

2k views

### Student's t-test for lognormally distributed samples [duplicate]

I want to understand whether the differences in the means of two independent samples (distributed lognormally) are statistically significant. In order to use Student's t-test, the data must be ...
51 views

### Is there a meaningful option to compute something like an effect size for Granger Causality?

I am running Granger Causality (GC) analysis using statsmodels.tsa.stattools.grangercausalitytests between several time-series pairs. The time-series pairs used for ...
500 views

### Variable is significant in multiple linear regression but not in t-test of the subgroups

I am getting different p-values for a variable in t-test and its coefficient in multiple linear regression so I am unsure which one to believe. As an example, my hypothetical dataset consists of ...
1 vote
42 views

### Comparison of two test metrics

I'm trying to compare two test metrics (Metric A and Metric B) to determine which one better predicts a delta value, which represents a Euclidean difference. I am unsure how to determining which ...
392 views

### Ideal test/p-value calculation for difference in means with small sample size and right skewed data? [closed]

I need to compare a response variable between two different groups. My control group has 205 observations while my comparison group only has 2 observations (unfortunately this is the nature of the ...
16k views

### Chi-square test or Z-test for comparing two proportions?

On this website the appropriate statistical test for comparing two independent proportions is described as a $Z$-test (a normal distribution is used to obtain a $p$-value). However, in R ...
235 views

### Estimating effects in the presence of a mediator

Suppose that one is interested to compare the effect of biological age versus "cognitive age" on a variety of outcomes. Cognitive age is measured by testing intelligence. The outcomes are a ...
31 views

### How to measure Statistical Significance for calculated Sensitivity, Specificity, Precision, Accuracy and f1 values?

I am trying to understand how to calculate one or more measures of statistical significance to display alongside metrics I've calculated from my data. Abbreviations I am using in the rest of this post:...
1 vote
279 views

### Doubts with P-value and Hypothesis in Permutation Test problem

I have to solve this problem for a stats course: An e-commerce company is testing a new design for a web page. The objective is to achieve at least a 2% increase in the conversion rate. An experiment ...
368 views

### How to calculate p-value to compare AUPRC for two models?

I've built two deep learning models and was looking for a way to compare the two models in terms of the area under the precision recall curve terms. I know that the DeLong method is recommended for ...
1 vote
274 views

### What do the p-values ​resulting from the SIMPER analysis in R represent?

I am doing a SIMPER analysis (in R) with the values ​​relating to individuals contacted in different reproductive seasons to see how the surveyed bird community has changed, and which species ...
553 views

### Comparing two sets of P values

I have two sets of P-values. Each set contains about 1500 P-values that are computed by different versions -i.e. slightly different algorithms- of the same software. These two versions used the same ...
21 views

### Why Do Different MANOVA Test Statistics Produce the Same Approximate F and P Values in R?

I am using the manova() function in R to perform multivariate analysis of variance on my dataset. I have tried using different test statistics such as Pillai's ...
Generally, I believe in giving exact p values rather than something like " p < 0.05". But, sometimes, the p value is extremely low, on the order, say, of $10^{-10}$ or even lower. Under ...