Skip to main content

Questions tagged [statistical-significance]

Statistical significance is a characteristic of a statistic viewed in light of a null hypothesis and a given significance level. It reflects whether the statistic belongs to the rejection region (is statistically significant) or the acceptance region (is not statistically significant).

Filter by
Sorted by
Tagged with
0 votes
0 answers
7 views

Measuring statistical significance of comparison results of two models with an imperfect classifier

I have a classifier with 87% accuracy for determining which of the outputs from two text generation models is closer in style to another sample text that is from some target author. Using this ...
doij790's user avatar
2 votes
2 answers
39 views

Is considering a specific distribution necessary before computing an average?

Many times, I compute averages of variables without considering distributions at all, and I use those computed averages to represent measures of variables in my data without mentioning specific ...
xabzakabecd's user avatar
  • 3,525
0 votes
1 answer
39 views

getting insignificant p-values but large effect size what should I do?

I used Qualtrics to conduct a survey and received 700+ responses. I also collected demographic data to disaggregate the data and observe any differences between groups. A couple questions used display ...
neybeh's user avatar
  • 1
3 votes
0 answers
21 views

How to identify hot spots in one-dimension

I am looking to identify stretches of a road along which a notably high number of accidents occur. My data can be represented as a two column table in which each row represents one accident, and the ...
Josh O'Brien's user avatar
0 votes
0 answers
7 views

statistical test to compare a single value (mutation number) with a set of mutation numbers

I have a single mutation value of a pathogen for a specific country that is 6. This same pathogen has different values for other countries as 5,7,8,9, and 10. How can I compare the statistical ...
Rubayetul Alam's user avatar
0 votes
0 answers
11 views

understanding fisheries model output tag and release

I am reviewing (for myself) a publication that has analysed data from an experiment to see what effect underwater seismic has on catchability of lobsters. de Lestang et al (2024) Fisheries Research ...
Andrew Halford's user avatar
0 votes
0 answers
20 views

test for differences in empirical cumulative distribution functions between groups

What test could I use to test for differences in empirical cumulative distribution functions between two or more groups? Note that I am not looking for a test two compare two ecdf (Kolmogorov–Smirnov ...
ucfaard's user avatar
  • 43
6 votes
1 answer
414 views

Related low p-values that do not meet statistically significant thresholds

If I were to have multiple related measures, each with "low" p-values that do not quite meet the threshold of significance, could the fact that multiple measures produce a general trend with ...
Jaxton Winder's user avatar
0 votes
0 answers
14 views

Significance of the coefficients of a multinomial mixed regression built with npmlt function

I fitted (using npmlt function) a mixte multinomial regression model which ran without errors. I expected that the outputs of the model include p-values that I didn't find. My need now is that someone ...
Romaine TCHEDJI's user avatar
0 votes
0 answers
21 views

Impact of variation in the total number of samples when performing multiple fisher exact tests

I have dataset of 2 time points. In each time point I have to compare the counts of different components at each position (number of positions =1000). I used fisher exact test to compare the if there ...
Marwah Al-kaabi's user avatar
0 votes
1 answer
26 views

Dickey-Fuller test statistical significance

I recently read about the Dickey-Fuller test. Firstly about the transformation from $$ y_t=\rho y_{t-1}+\epsilon_t $$ to: $$ y_t-y_{t-1}=(\rho-1) y_{t-1}+\epsilon_t $$ I assumed it is to get the ...
Tomer Gigi's user avatar
2 votes
0 answers
31 views

Problem regarding False discovery rate correction for multiple comparison

I am conducting research to find association of a gene with a disease. It's a case-control study. I have divided the cases into groups A and B based on genotype and controls into groups C and D. I ...
Confused_biologist's user avatar
0 votes
0 answers
10 views

Significance of factors/regressors on compositional data

Let's assume that I have composite data, similar to the "ArcticLake" example dataset in the DirichletReg package in R. Now, I would like to know whether various numeric variables and factors ...
Anti's user avatar
  • 159
4 votes
2 answers
342 views

confidence interval and rejection

Assume I have a data sample $X_{1}, \dots X_{n}$ from a normal distribution. Then, we consider the following hypothesis $$ H_0: \quad \mu = \mu_{0} $$ and alternative $$ H_1: \quad \mu \neq \mu_{0} $$ ...
ABK's user avatar
  • 676
0 votes
0 answers
8 views

using ranksum test within permutation test scipy

I am a bit confused with the definition of the permutation_test function provided by scipy. the following is what I wrote to calculate the p and null distribution ...
HappyDuppy's user avatar
1 vote
0 answers
20 views

Testing the difference between two Root Mean Square Error values for statistical significance [duplicate]

I would like to compare the predictive power of 2 models. The models are meant to model count data and respective probabilities. I am using two metrics as means of comparison: Root Mean Square Error ...
Astral's user avatar
  • 133
0 votes
0 answers
15 views

Help determining sample size without standard deviation

I need some help optimizing our website intercept survey sample sizes, but our online platform only provides us with the means (often mean of 5pt ordinal scales), sample sizes and denotes statistical ...
PoolHound's user avatar
3 votes
1 answer
49 views

Testing for difference in trends of binary-parameter proportion/fraction versus continuous parameter between two (or more) different populations

I have three samples of objects: sample X (total = 138), sample Y (same basic class as X, but located in a different environment; total = 186), and sample Z (total = 996). For each object, I have a ...
Peter Erwin's user avatar
0 votes
0 answers
8 views

Measuring statistical difference across two distributions of attention scores

For a study I'm conducting, I have two distributions of attention scores generated via a trained transformer model. I'm plotting the two distributions and I would like to run a statistical test to ...
James Arten's user avatar
2 votes
0 answers
28 views

What is the correct sample size for the purposes of this test?

$X_i$, $i = 1, \ldots, n_X$, are i.i.d. from some distribution; and $Y_j$, $j = 1, \ldots, n_Y$, are i.i.d. from some other distribution. Consider all pairs $(X_i, Y_j)$. The number of pairs is $n=n_X ...
Jessica's user avatar
  • 1,251
0 votes
0 answers
11 views

Experimental Design: Selecting value of $n$ given desired width of credible interval

Note that this is a cross post from here. I realize this is probably a better space Suppose I have $n$ IID Bernoulli trials with $k$ successes. Assume that as a prior we are assuming that $P(\theta)$ ...
wjmccann's user avatar
  • 113
8 votes
4 answers
219 views

Citations explaining the problems with two-stage statistical testing

I am writing a review about analyzing data sampled from lognormal distributions, and I want to explain the problem with first running a normality test and using that result to choose which test to run....
Harvey Motulsky's user avatar
0 votes
0 answers
30 views

How to measure Statistical Significance for calculated Sensitivity, Specificity, Precision, Accuracy and f1 values?

I am trying to understand how to calculate one or more measures of statistical significance to display alongside metrics I've calculated from my data. Abbreviations I am using in the rest of this post:...
Natalia's user avatar
0 votes
2 answers
45 views

Long term monitoring data statistics help [closed]

I am still learning statistics and I have some long term monitoring data taken from soil samples in two different forests over 5 months. Soil chemical parameters such as pH and redox, carbon and ...
martyn's user avatar
  • 21
8 votes
6 answers
571 views

Is it better to perform multiple paired t-test or One-Way ANOVA test

I have 7 patients with blood measurements from two different tissues and at two different oxygenation states. We first took measurements from tissue A and B (Normoxia) and then we administrated oxygen ...
user161260's user avatar
2 votes
0 answers
30 views

What makes a statistic valid for monte carlo simulation?

A while back I was reading Garland et. al. (1993) about studying whether two groups of animals, say herbivores and carnivores differ in their mean value for some trait, like the amount of territory ...
A Friendly Fish's user avatar
0 votes
0 answers
24 views

how to measure significance of improvement between two methods during modeling

Hello everyone I have two methods for developing models. I use a set of input features (set1) for training model1 and another set of input features which have (set1 features + extra set2 features). I ...
Rhea Bedi's user avatar
4 votes
0 answers
64 views

Knowing when to stop benchmarking

I have a basic question that I'm sure must have been asked before, but I don't know the words to Google... I do a lot of benchmarking software, ie comparing two different implementations of the same ...
Matt Parlane's user avatar
0 votes
0 answers
11 views

Calculating significant difference between several averages of the same samples

I currently have a dataset consisting of 8 measurements taken around a cylindrical core sample at intervals of 90 degrees to each other, 2 measurements per point (observing the core face on these ...
Sacha's user avatar
  • 1
2 votes
1 answer
75 views

Predicting cost for several options and then choosing the cheapest option

I created an ML model that predicts the cost of shipping an order in our online store. For each order placed, there is a variable number of methods (up to 10) available to ship. So for one order, ...
rimsky's user avatar
  • 23
0 votes
0 answers
21 views

How to properly compare a single diagnostic test in two subpopulations

Suppose I have a diagnostic test with a binary response. I have performed this test on some population. Now, I split this population in two subgroups (e.g. male/female), and would like to check if ...
Alb's user avatar
  • 115
1 vote
1 answer
78 views

Different p value from Wilcoxon and ANOVA

I got a different $p$-value from a Wilcoxon test and an ANOVA test. When I use the Wilcoxon test, the $p$-value is less than $0.05$: ...
bibi's user avatar
  • 11
5 votes
1 answer
64 views

Do they use some Bayesian-frequentist amalgamate in astrophysics?

Below is a figure (highlighting is mine) from Madhusudhan et al. (2023). It caught my attention in a recent video by Becky Smethurst, where she explains some more context (but that's not necessary for ...
Durden's user avatar
  • 1,350
1 vote
1 answer
24 views

Evaluating a classifier's performance on different groups of subjects

I have developed a binary classifier that predicts whether a subject is injured or healthy. I am interested to know whether my model performs better on certain groups of subjects than on others. For ...
Roy Phillips's user avatar
0 votes
0 answers
22 views

Statistical Significance Testing for Nested Cross-Validation in ML Experiment

I am currently working on an ML experiment where I use a nested 5-cross validation procedure and obtain a NDCG@10 scores for each test user. I am comparing 6 different ML algorithms and have data for ...
Bernhard's user avatar
  • 101
0 votes
0 answers
38 views

Comparing sample means to standard measurements

I am trying to best address how to determine if a sample measurement is statistically different than a standard. If I measure any variable (lets say mass) for a sample 1000 times to determine an ...
Dan's user avatar
  • 1
5 votes
1 answer
252 views

Is there an expression that can be used to evaluate the threshold that characterizes a substantial difference between sample means for t-test?

I'm currently performing a paired t-test for the difference between sample means. I have an interest in determining if there is a way to obtain a scalar value for a threshold that quantifies whether ...
Justin's user avatar
  • 53
0 votes
0 answers
56 views

gene total volume test

1, I want to compare the FGF gene total expression level (sum level from all the cells) in two groups of cells. Suppose group A has 500 cells and group B has 100 cells. 2, gene FGF level in the two ...
coding_something's user avatar
0 votes
0 answers
54 views

Logistic regression only cares about the $E[X|Y]$ for prediction

I was reading this blogpost about how logistic regression and Naive Bayes with gaussian features are equivalent This led me to think that at the end of the day, logistic regression only cares about ...
Alberto's user avatar
  • 1,217
0 votes
0 answers
12 views

How to test whether the values in one matrix are significantly smaller than in another matrix?

I have a set of proteins for which I have predicted structures using two distinct methods. For each protein, both methods have generated 5 different structures. My goal is to compare the stability of ...
Amara Deschutter's user avatar
2 votes
1 answer
40 views

Minimum Survey Respondents for Hypotheses I want to answer

I am an eager statistics noob, so apologies on the basic questions. I am looking to send out a survey to a population that is large (people with diabetes who take insulin in the US) (~ 8.4 million) ...
Christine's user avatar
3 votes
1 answer
154 views

Paired t-test of two algorithms

I want to compare the speed of two algorithms. However I'm having troubles interpreting the results. Is my approach correct? Null hypothesis: The algorithms have same speed Alternative hypothesis: ...
popcorn's user avatar
  • 143
0 votes
0 answers
23 views

Test of difference of Coefficients of a same regression [duplicate]

Sir I have run a regression, where I estimated effect of two independent variables A and B on an outcome variable C. How can I compare the regression coefficients of A and B to estimate whether the ...
adarshad's user avatar
0 votes
0 answers
16 views

Statistical test for variability among more than one group

I am very new to statistics and still learning. Below are the funnel graphs which shows the effect/percentage of polypharmacy(patients with two or more drugs) against practice/hospital population size....
Usman YousafZai's user avatar
0 votes
0 answers
16 views

How to report few-shot accuracy for LLMs?

I am comparing three prompting techniques in LLMs to check which one is best. All prompting strategies include three examples for in-context learning (few-shot only, no fine-tuning). If I do greedy ...
Jader Martins's user avatar
2 votes
1 answer
33 views

Indirect effect present in separate models but disappears in full mediation analysis

I am wrote my thesis and looking forward to defending it. So my question is about indirect effects through a mediator. I have the following situation: $X$ (parental involvement in education) doesn't ...
Viktorija Urmanavičiūtė's user avatar
0 votes
0 answers
65 views

Interpreting by dividing up results of AB test

at my company we are doing AB tests (with 95% confidence) for features of our game (mobile app, hyper-casual game, Global scale). After the tests had ran its course, we have a practice of dividing up ...
Thinh Vuong's user avatar
2 votes
0 answers
29 views

A/B Test for CTR without user-level data

Suppose I want to show two different ads and compare which one generates more clicks. By design I know that each user will see one of two ads at random every time s/he visits an external page where ...
ssrg's user avatar
  • 21
4 votes
2 answers
191 views

Question about combined relative risk interpretation

I am looking at a review of the effects of cigar smoking, and I am having trouble interpreting some of the results. Study linked here: Study Specifically I have questions about table 3, below: In the ...
user5597411's user avatar
0 votes
0 answers
17 views

Prove Decreasing Cross Entropy of outputs with Decreasing KL divergence of inputs

I am trying to prove the inequality $H(gt, y) > H(gt, y_1) > H(gt, y_2)$, given that $D_{KL}(x, x_1) > D_{KL}(x_1, x_2)$, where $y = f(x)$, $gt$ - ground truth, $D_{KL}$ - KL divergence, $H$ ...
user412867's user avatar

1
2 3 4 5
130