Questions tagged [statistical-significance]
Statistical significance is a characteristic of a statistic viewed in light of a null hypothesis and a given significance level. It reflects whether the statistic belongs to the rejection region (is statistically significant) or the acceptance region (is not statistically significant).
6,500 questions
0
votes
1
answer
45
views
Is there a standard way to quantify & test the benefit to one covariate by including a second covariate?
For a linear model (or glm), is there a standard statistic (and associated test) to quantify whether inclusion of a covariate increases the strength of the association between another covariate and ...
- 225
0
votes
1
answer
47
views
Bradley's liberal criterion
I'm reading an article where the authors utilized Bradley's liberal criterion to estimate the robustness of the F statistic in the context of post hoc tests.
The problem here is that they said the ...
4
votes
4
answers
544
views
How to permute p-values?
Recently, my colleague encountered a problem while working with a dataset, df_a, which contains gene data. Each row represents a ...
- 153
0
votes
2
answers
53
views
Suitable statistical test
I have 2 groups - people that were ever active on my website and people who were never active. Activity is a metric based on some actions - both groups of people visited the website.
Each group used a ...
0
votes
0
answers
16
views
Estimate SD[Z=X*Y] which one is correct?
Given we have a sample of two variables X and Y with sample size n. I want to calculate the standard deviation of Z = X*Y. I don't know which of the two options bellow are correct?
Option 1:
Simply ...
- 193
0
votes
0
answers
22
views
Model comparison and experimentation for a thesis result
We are conducting a study to compare the accuracy of two computer vision models:
Model A: Trained on a non-augmented dataset of 11,200 real-world images.
Model B: Trained on an augmented dataset ...
- 101
0
votes
0
answers
117
views
Which significance test to use for a difference of means? How to be sure?
[Updated] I have two data sets containing all official recorded games played in the USA for this one sport. One data set is for the women's teams and the other for the men's teams. I'm trying to ...
0
votes
0
answers
51
views
Am I going about statistical significance testing for weighted categorical survey data the right way?
I created a (fictional) dataset that contains individuals' car colors and their level of agreement to the question "I enjoy driving fast". The survey includes three car colors (red, black, ...
- 1
1
vote
0
answers
36
views
Accounting for Retention Patterns in YouTube thumbnail A/B/C Testing
I am using the YouTube thumbnail A/B/C testing feature and encountering results that are implausible – I am trying to figure out why. One theory is that YouTube is not taking retention into account in ...
- 11
1
vote
2
answers
80
views
A test to check whether two sets of proportions are _not different_
Is there a test to check whether two small sets of proportions are significantly not different?
For example, consider the two sets of proportions
...
- 777
2
votes
1
answer
41
views
Is power of a test lower bounded by the significance level? [duplicate]
Here's an example that makes me ask this question. Let the significance level be $\alpha$.
Suppose we have a normal distribution with a known standard deviation $\sigma = 1$.
$H_0$: The mean of the ...
- 23
0
votes
0
answers
23
views
Determining number of observations be within confidence threshold
I have a model which has an accuracy of $A$. It makes a prediction with $Y$ confidence about a number of samples. Samples are observed many times, but not all are observed equally. So my data looks ...
- 51
1
vote
1
answer
57
views
What level of statistical significance to choose for test of equaility of seemingly unrelated regression coefficients?
It is possible to compare effects from models estimated as seemingly unrelated [1] [2] [3]. DVs may be related, IVs may be related, or both may be true. In any event, analysts test for equality of ...
- 326
1
vote
2
answers
78
views
Does every finding in medical statistics need to be validated in an independent dataset?
I am performing a medical statistics analysis about 500 independent variables and 1 dependent variables. The sample size is 1200. The relation between the variables and outcomes are weak (say, diet ...
- 13
0
votes
0
answers
12
views
Appropriate statistical significance test for categorical independent variables with binary dependent variable
I am trying to determine whether a language prediction model predicts significantly more accurate sentiment values than other language prediction models.
There are three language prediction models ...
- 13
2
votes
1
answer
50
views
AB test design in eCommerce - group split per user and statistic aggregation per item
Is it statistically correct/viable to run an A/B test where the split into groups is per user (randomly assigned) and the statistic is aggregated per store item?
Lets narrow down the issue into a ...
- 23
0
votes
0
answers
51
views
P-Value correction across/within multiple datasets
The analysis in my work encompasses a multitude of hypothesis tests and I am not quite sure whether, or how, to control increased false-positive probability. Outside of post-hoc analysis following ...
0
votes
0
answers
23
views
T distribution for testing significance of Spearmann correlation coefficient
I look for a proof that one can use statistic $t=r_s \sqrt\frac{n-2}{1-r_s^2}$ which follows t-distribution with n-2 d.f. for testing whether Spearmann correlation coefficient is non equal zero. I ...
1
vote
2
answers
56
views
How do I interpret different results from simple regression, multiple regression, moderation, and mediation?
I'm currently doing a multiple linear regression analysis on Spirituality and Forgiveness to see how it affects Life Satisfaction.
If I put the IVs individually into linear regression they both show ...
- 11
11
votes
4
answers
647
views
Generally, are we supposed to remove all nonsignificant paths in an SEM analysis?
Given that we favour parsimony in science, should we remove nonsignificant paths that are nonsignificant if they do not result in significantly worse model fit?
After we remove the nonsignificant ...
- 331
2
votes
0
answers
24
views
Testing significance of difference between distance matrices
In microbial studies one often looks at beta-diversity in terms of a PCoA decomposition of a distance matrix between the samples, e.g.:
(image source)
This usually serve to draw various conclusions (...
- 4,487
3
votes
1
answer
50
views
Cumulative incidence function - compare time at which a specific cumulative incidence is reached
I have survival data and I've plotted cumulative incidence functions stratified by treatment status (with age as the time scale). I used Gray's test to compare the overall cumulative incidence ...
- 75
0
votes
2
answers
94
views
significance of an observation in a small sample
In a population of 10000 individuals, I test 30 individuals and find that all of them have a disease. Based on this sample size, can I carry out some statistical test to find what proportion of the ...
- 13
5
votes
1
answer
228
views
In SEM, is there any point in interpreting path coefficients if the model shows poor fit in the first place?
If there were some path coefficients that were significant but the whole model shows poor fit, can I still use the result from the significant path?
By extension, if the model evidenced poor fit in ...
- 331
1
vote
0
answers
50
views
Best test to evaluate vote significance [closed]
What is the best test to evaluate a vote's significance and why?
This article evaluates the US Supreme Court voting system and the voting system of the congress and says both fail significance tests ...
3
votes
1
answer
75
views
What model to use for multivariable analysis with small sample size
I have a small sample size of around 350 of which around 20 or so had an event. I have around 12 variables that might be associated with the event and hoping to do multivariant analysis, some of which ...
- 31
8
votes
5
answers
938
views
Can one use the p-value to perform hypothesis testing instead of comparing the test statistic to the critical value at a given significance level?
In hypothesis testing, if a question states the level of significance, which is alpha, then does it necessarily mean that we have to use the classical method? (Means finding critical value and then ...
0
votes
2
answers
28
views
Statistical test of ensemble averages
Machine learning models are evaluated by their predictions, producing a single-value score (e.g. mean squared error). Machine learning models are initialized randomly and that affects their score. ...
- 11
0
votes
0
answers
25
views
Significance of coefficients in GARCH-MIDAS model
I used the GARCH-MIDAS model to check the relationship in stock volatility and epu, but the some countries’ $\alpha$ (the long term volatility parameter) is 0, and the p value is 1. The condition of ...
- 1
4
votes
1
answer
72
views
Statistical significance of multiple models over multiple datasets
I have a table of mean performances of 6 models over 10 datasets. While I've seen tables like these in research papers, they always display significant results in bold. What I'm not sure of is how ...
0
votes
1
answer
44
views
P-value changes when invert coding of binary outcome in Poisson GEE with log link but not logistic GEE or OLS
We are running regression models with a binary outcome (Yes/No) on panel data. The outcome is very common (93.6%=Yes, n=3961 of 4231 observations). We are using GEE models to account for the multiple ...
16
votes
2
answers
1k
views
Does every variable need to be statistically significant in a regression model?
I recently fit a regression model (ARIMAX) in which some variables (3) were statistically significant and some were not (1). I removed the statistically insignificant variables and refit the model, ...
- 315
0
votes
2
answers
35
views
Structural break detection in export data
I'm looking at Norwegian export data to try to detect sanctions evasion by Russia. Sanctions evasion takes place by the export of goods to third-party countries that in turn re-export the goods to ...
6
votes
4
answers
626
views
Statistical test for count values
I have 4 stories that I have the word count and the count for how many plural words are in each:
Story
Num Words
Plurals
1
356
45
2
273
23
3
303
28
4
289
42
I am wondering if it is possible to ...
- 63
2
votes
0
answers
31
views
How Does Autocorrelation in a Feature Disrupt Linear Regression? [closed]
I am running a linear regression model, and one of my features shows autocorrelation. I’m trying to understand how this affects my model and what the best way to address it might be.
From what I ...
- 21
1
vote
0
answers
33
views
Bayesian A/B Testing
I've been reading a lot about A/B testing and I understand the basic frequentist methods. I'm having a hard time with the Bayesian approach and was wondering if i could get an explanation of exactly ...
- 31
7
votes
3
answers
572
views
I have two different statements for the t test and the single tail test
I want to check whether the values of a group (ct2 are greater than the values of ct1). To do this, I have applied a T test which has the following results.
...
- 185
4
votes
1
answer
86
views
Creating a dummy variable when the continuous variable is equal to 0
I'm actually trying to find the best explanatory variables in order to estimate the probability of deafult of the counterparties of my portfolio. After defined the Long List of variables, I'm testing ...
- 41
3
votes
1
answer
83
views
Selecting statistical test for significant difference between groups from count data with zero values
I wish test significant difference between groups from count data with zero values. I only want to compare groups with the same color (grey, yellow and blue). Some groups have only zero values which ...
0
votes
0
answers
17
views
How to automatically filter out models fitted with auto.arima function based on bad residual properties?
I am looking for an easy way to filter out "bad" models fitted by auto.arima function.
These are SARIMA models with different parameters and by a "good" model I mean model with ...
- 291
0
votes
1
answer
19
views
I Have a rank-order list of healthcare priorities which i assessed using a survey between doctors in two different countries
I used a survey to assess the rank order of certain healthcare priorities between healthcare providers in two different countries. I have tested the man rank-values attributed to each priority using ...
1
vote
1
answer
60
views
Correlation between two large binary arrays
Imagine I have two large arrays of 0's and 1's representing locations along a one-dimensional spatial domain, where:
$$
X = \text{Locations where the error is high},\\
Y = \text{Locations where data ...
- 150
0
votes
0
answers
30
views
How to properly handle hypothesis testing in the case of no rejection
I am learning about hypothesis testing and am wondering how to properly test a hypothesis in the following case. I am rather new to the platform, please let me know if I made a mistake in my question (...
2
votes
1
answer
27
views
Appropriate test for RNA binding data
I have the data on RNA binding by a protein. The dependent variable is the fraction of bound RNA, and the independent variable is protein concentration. I have such data for one reference RNA and 8 ...
2
votes
2
answers
45
views
How to prove an inference cannot be drawn from an irrelevant test?
Is anyone aware of good sections in statistics literature or textbooks that explain why it is essential to make sure that the tests you are running are relevant to the question you are investigating? ...
1
vote
1
answer
88
views
How to calculate and compare the apparent likelihood of an interaction between groups of different sizes
I need to compare variation in virus transmission within and between three differnt sized age groups: children (N= 20,000), adults (N= 60,000) and older people (N=20,000). So, within the population, ...
- 61
0
votes
1
answer
30
views
How to run a statistical test to look for over-enrichment of specific genes, across three different phenotypes, row wise? [duplicate]
This is my contingency_table:
...
- 29
1
vote
1
answer
42
views
Alternative for three-way ANOVA - 2 out of 3 groups have non normal distribution in their categories
I am analyzing capacity values from an aging test of 24 battery cells, which have three categories:
Battery Type: they can be 29V or 33V
Storage Temperature: 10°C, 20°C or 30°C
Storage SOC: 30% or ...
- 13
3
votes
2
answers
76
views
When to Apply P-Value Corrections for Multiple Testing: A Multi-Outcome Study Example
Suppose I want to examine the association between a total score and independent binary variables (e.g., sex) and multi-categorical variables (e.g., grade). I conducted a t-test for the binary ...
- 71
0
votes
2
answers
27
views
Chi Squared Test of Association, Feature Engineering and Selection book, Chapter 2
In feat.engineering/stroke-tour by Kuhn & Johnson, there's a contingency table as below:
But I tried computing the chi-squared myself by hand and with a calculator and get 0.9818 with p-value ~0....
- 103