New answers tagged statistical-significance
2
votes
Accepted
Building a bootstrapped distribution to test reliability of empirical data
This sounds more like a candidate for a permutation test.
Relevant for the conclusions of the study is whether a decrease in this number took place across generations.
So your hypothesis is that ...
Tim♦
- 113k
0
votes
Interpretation of p-value near alpha level
Frame challenge: Focus on the data you gathered and on weighing hypotheses against each other in metrics meaningful to your domain, not on the cult of the $p$-value.
The cult of the $p$-value is a ...
1
vote
P-value under 5% but power under 80%
If you observe a p-value equal to the alpha level, then you have roughly $50\%$ power for a true effect that equals the observed effect. Observations that are closer to the null hypothesis will not be ...
- 46.5k
1
vote
P-value under 5% but power under 80%
Possible situations abound. Here a few:
Suppose you are testing $H_0: \mu = 50,$ against $H_a: \mu \ne 50$ at significance level $\alpha = 0.01 = 1\%,$ using $n = 10$ observations from a normal ...
- 49.9k
3
votes
Accepted
How to assess economic significance in a log-log OLS model?
Statistical and economic significance are not the same concept. Broadly speaking, statistical significance in a regression tells you if a variable has an effect on the outcome variable (depending on ...
1
vote
Interpretation of p-value near alpha level
One consideration that is missing from the discussion is the problem when you have more than one test. As your p-value defining significance increases there is a concomitant increase in the chance ...
- 11
6
votes
P-value under 5% but power under 80%
Power depends on the $\alpha$-level, which you set, not on the p-value (which you calculate).
If you do a power calculation and determine that you have $80\%$ power to detect the difference of ...
- 31.2k
6
votes
Interpretation of p-value near alpha level
I will also add to Jeremy Miles' answer that makes a number of valid points. (I wrote initially that "I disagree with the claim that the reason that statistical significance testing doesn't seem ...
- 11.5k
2
votes
What is the impact of duplicate data on the variance of regression coefficient?
The coefficients themselves will no change.
Imagine you perform the analysis on the first dataset, and plot the regression line with the datapoints around the regression line.
Now what would happen if ...
- 570
4
votes
Interpretation of p-value near alpha level
I would add to the excellent answer of Jeremy Miles by saying that how you treat your p-values also strongly depends on what you want to do with them. They get a bad reputation (and rightfully so), ...
- 241
1
vote
Permutation testing: Which variable should be shuffled?
With respect to a simple correlation between a single X and a single Y, you could shuffle either. That said, the p-value isn't "how often the resulting correlation coefficient exceeds the ...
30
votes
Accepted
Interpretation of p-value near alpha level
There are two different approaches to interpreting statistical significance - the Fisher way, and the Neyman-Pearson way. We smush these together (into what Gerd Gigerenzer has called a 'bastardised ...
- 14.4k
3
votes
Accepted
Statistical test to check if data come from two different lines or just one
This is a standard application of an interaction term in a model.
With x as continuous-valued predictors and y as your outcomes, let S represent which set a data ...
- 62.4k
3
votes
Accepted
Usefulness of KS tests and other similar distribution comparing tests
This is one of those ideas that starts out sounding great but winds up being less helpful than one might hope.
For instance, just because KS (or a similar test) says that the feature has a different ...
- 31.2k
1
vote
Accepted
How many values should one validate to achieve a desired confidence?
There are a few ways to think about this.
If you want to use the confidence interval approach, then you should also look at the "finite population correction" (Googling for that phrase ...
- 46.9k
0
votes
Is confounding a source of Type I errors?
First, let's talk about what you mean by "confounding". If you mean common causes of the treatment and outcome, then there is no confounding in randomized controlled trials. If you mean the ...
- 21.9k
1
vote
Is confounding a source of Type I errors?
Confounding can result in the identification of spurious, non-causal relationships that you don't care about, but I would not characterize these as Type I errors since there is a real statistical ...
- 5,743
5
votes
Accepted
Unusally high F value ANOVA
With sample sizes as large as those you will almost always find that the statistical test gives very low p-values. Divide those SD values by the square root of the sample sizes to see the estimated SD ...
- 11.2k
0
votes
How to compare variables with data from multiple experiments?
You could treat each repetition of your experiment as a i.e. you probably have a randomised complete block design. If you have replicated the experiment in each repetition then you can estimate block ...
- 113
1
vote
Wilcoxon.test in R will not calculate exact distribution due to ties (scipy.stats.wilcoxon will)
I don't have enough reputation points to add this as a comment, so although not an answer, hope it does help.
I've adjusted your code a bit for convenience:
...
- 11
1
vote
Accepted
How do I test that two samples don’t differ significantly by their demographics?
You can run tests to check that demographic variables are balanced. However, it's not recommended to do so for a randomized study. As you say, you have to trust the randomization. What's recommended ...
- 2,156
1
vote
Which statistics test should I use?
Comment:
A suggestion has been made to do a chi-squared
test on your $2 \times 6$ table of counts to see if the distributions for High and Low are
different. That is one reasonable possibility among ...
- 49.9k
7
votes
How to report a P-value?
When is it appropriate to give the exact P-value, instead of writing e.g. P<0.05 (also in case of non-significant P-values)?
As a general guideline you want to convey as much information about ...
- 46.5k
1
vote
How to report a P-value?
In practice, people report the exact p-value then declare whether the hypothesis test is statistically significant. Most statisticians don't advocate this. if the goal is hypothesis testing then ...
- 53k
0
votes
Statistical analysis (comparison) of time course experiments
Is there a tried-and-true statistical method to get a single p-value to determine whether two such curves are significantly different?
Unless you have some theoretical form for the change in signal ...
- 62.4k
3
votes
Why do discrete choice models (such as MNL) not require test set?
Multinomial logistic regression can and often does consider out-of-sample performance. For instance, LeCun (1998) applied multinomial logistic regression to the pixels of the MNIST handwritten digits, ...
- 31.2k
1
vote
Is a p-value of 0.04993 enough to reject null hypothesis?
The 0.05 threshold is a hurdle that you have set for yourself in order to enforce a degree of self-skepticism about your alternative hypothesis. It somewhat weakens that self-skepticism if you change ...
- 48.5k
1
vote
Is a p-value of 0.04993 enough to reject null hypothesis?
In light of the assumptions of your model, you should reject the null because dichotomizing claims based on hypothesis tests have clear epistemological and pragmatic functions. But never forget that: ...
- 11
0
votes
How to determine which variables are statistically significant in multiple regression?
Yes, you should look at the last column which contains the p-value parameter.
Usually, we consider that if p-value < 0.05 for a certain variable then it is significant and has some relationship ...
- 230
2
votes
Correlations significant at individual level, but not at group level?
See the answer at Correlation: average observations Vs observations: When you correlate averages, it will tend to be stronger because averaging reduces noise/variance.
But, since the number of pens (...
- 64.9k
1
vote
How to calculate number of participants required to compare mean scores on a questionnaire between two groups?
It is a pretty old question here but since I know of, and made use of a straightforward way of calculating this I ‘d like to share it depending on whether you wish to have a double sided or a single ...
- 11
0
votes
References: t-test and Chi-squared test can be conducted with unequal sample sizes
Many intermediate-level applied statistical texts have the warnings mentioned in my comment. (One example, among many, is the text by Ott & Longnecker.)
Use Welch, not pooled t test. Here is an ...
- 49.9k
0
votes
How can I compare robust linear mixed models and get the p-values, F statistic, AIC and BIC?
This answer comes late to the OP but I found this post trying to get the answer to the OP's question and then later found part of the answer. @Wagliss is correct that you cannot obtain an AIC or BIC ...
1
vote
Collinearity problem
Probably yes, we can suppose that BMI and weight are highly correlated. You can verify it by performing a correlation between BMI and weight and checking the resulting coefficient, but as in previous ...
- 29
0
votes
Simple question on statistical comparison of grand/pooled means
I infer that your question is what can be done about this. You need to take advantage, carefully, of the numbers of relevant observations going into each "mean of means" and of the ...
- 62.4k
0
votes
Can I compare just some grades to see if they are statistically significantly different from one another?
I think this is a misunderstanding of what statistics does. You cannot in any meaningful sense do a test to state that student X's grade in one test is significantly different from student Y's. ...
- 11.9k
1
vote
Accepted
Is a huge F-statistic OK?
Let $n$ be the sample size.
Define $RSS$ to be the residual sum of squares for the model, and define $TSS$ to be the total sum of squares. That is:
$$
RSS = \sum_{i=1}^n (y_i - \hat y_i)^2\\
TSS = \...
- 31.2k
1
vote
Accepted
When to drop correlated features?
"Significant correlation" would usually mean that you tested a null hypothesis that $\rho=0$. Depending on your sample size, such correlation may still be quite close to zero. Why would you ...
Tim♦
- 113k
0
votes
difference in proportion in multiple sets
To be clear, your table suggests strategies, but the 'round' numbers would be unusual for real data. You'd need data consisting of counts of punches from boxing matches between boxer A and each of ...
- 49.9k
1
vote
Significance Testing of Difference in Variances of Autocorrelated Time Series Data
There's a problem here: Testing the difference in variances when?
The time series you've shown are clearly non-stationary. What this means is that there is a pattern over time. More formally, a time ...
Top 50 recent answers are included