New answers tagged hypothesis-testing
3
votes
Calculating statistical significance of gender discrimination in a CS class
As well as the issues already raised in the comments and the answer - if you were motivated to conduct this test just because you noticed the gender difference then you are HARKing (hypothesizing ...
- 1,396
0
votes
Proof that Sargan test statistic is distributed $\chi^2$
I will use your notation and write $M_A = Id - P_A = Id - A^T (A^T A)^{-1} A$ for the projection onto the orthogonal complement of $A$.
Let's assume all k regressors in $X$ are endogenous, i.e., $k=m$....
2
votes
Calculating statistical significance of gender discrimination in a CS class
Your calculations are correct, as long as assumptions are correct (which may be not the case, as noted by Greg Snow in the comment section). However, one important problem is that this is quite a ...
- 2,386
6
votes
Common ways of p-hacking in scientific literature?
A typical approach to answer such a question is to look it up at Wikipedia which has a broad article on this topic https://en.m.wikipedia.org/wiki/Data_dredging following the links in that article, ...
- 68.3k
0
votes
Accepted
Hypothesis testing in moderation analysis after Propensity Score Matching
What the code does is compute the log risk ratio (LRR) of the treatment for each level of the moderator (male). The first call (...
- 30.2k
2
votes
Accepted
Why does the score test work for values longer in the tail that have a small log-likelihood derivative?
Remember that we're differentiating the log-likelihood. If $L(\theta;x)$ goes to 0 as $\theta$ moves away from the peak, then $\log L(\theta;x)$ will go to $-\infty$.
Try sketching the log-likelihood ...
- 2,088
0
votes
Comparing pre and post treatment data - best statistical test?
I typically reference one of these. Save the ol' brain power for the programs.
1
vote
Tests that Quantify Deviation from Null Hypotheses
I don't think you want a test at all. A statistical hypothesis test tells you whether you should a) Reject the null or b) Not reject the null. That's all. That's just what tests do.
And the results ...
- 104k
2
votes
Why the probability of rejecting the null hypothesis tends to 1 in this case?
Sort of, but not quite: $\hat\mu$ being exactly zero isn't needed and you've left out some important information.
Consider the distributions. If $\mu=\delta$, then $$\sqrt{N}(\hat\mu-\delta)\stackrel{...
- 33k
2
votes
Accepted
How could Lilliefors use Monte Carlo if the estimand is not distribution-free?
shouldn't have Lilliefors done a Monto Carlo numerical estimate for all kinds of $\mu$ and $\sigma^2$?
If the $X_i$ are iid Gaussian variables, then $E$ is independent from distribution parameters $\...
- 68.3k
0
votes
What is the best test to use?
Multiple Regression is something you could do. After doing Exploratory Data Analysis with scatter plots and such you can decide which specific regression model you should go with but basically you ...
- 47
1
vote
How do we know the distribution of regression coefficients
OLS and MLE estimates are "sample means" -of some random variable.
For example, for OLS we have
$$\sqrt{n}(\hat \beta_{OLS}-\beta) = \left(\frac 1n X'X\right)^{-1}\cdot \sqrt{n}\left(\frac ...
- 56.5k
7
votes
How could Lilliefors use Monte Carlo if the estimand is not distribution-free?
It might not be distribution-free but for a given sample size it is location and scale free.
The internally standardized joint distribution of the observations does depend on the form of the ...
- 275k
2
votes
What is the best test to use?
My advice for beginning your explorations: Forget about statistical tests, and graph the data in various ways with as much detail as is reasonable.
- 19k
0
votes
Best test for comparison of temporal counts
It very much depends on what exactly you mean by "closely related".
One fairly obvious choice is correlation. This can be dangerous with time-series type data and lead to odd findings (e.g. ...
- 104k
0
votes
Understanding the significance level of a confidence interval
It just indicates that if you repeat your experiment 100 times (e.g., draw random samples of size n from the population and construct the ...
- 667
0
votes
Is there a multivariate two-sample Kolmogorov–Smirnov test?
For a practical application of the multivariate KS distance in high dimensions, see my article New Python Library to Evaluate AI-generated Data and Compare Models. I actually created a Python library ...
1
vote
Why is it possible to get significant F statistic (p<.001) but non-significant regressor t-tests?
While most of existing answers pointed out the multicollinearity is one common cause of this paradox, it seems that none of them demonstrated why it is so from the mathematical perspective. This ...
- 14.6k
1
vote
Two sample proportions: How to test the hypothesis that the difference in proportions is less than a specified threshold?
You say that your null is that the difference in conversion is at least 1 percentage point, but in the comments you also agree that you suspect the change is at least one percentage point. Those two ...
- 32.2k
2
votes
Transitivity of Wilcoxon signed-rank test
Does this meet the conditions of a counterexample? Or have I misunderstood something somewhere?
...
- 275k
5
votes
Accepted
Transitivity of Wilcoxon signed-rank test
Update: I'm no longer convinced that the ordering is transitive, but it is still described by the pseudomedians. It's just that when you don't have a location shift the pseudomedians need not be ...
- 33k
1
vote
Does the term “bootstrap” comprise Monte Carlo samples of null models a.k.a. “surrogates”?
For whatever it’s worth, I stumbled upon a publication that is essentially about this, namely namely Jason H. Moore, Phys. Med. Biol. 44, L11 (1999) (letter to the editor). The author notes that the ...
- 2,261
1
vote
How to do a pairwise test of each regression model coefficient over groups?
You can add the binary grouping variable (G) as a predictor to your model together with its product with each predictor variable X (moderated regression), for example, G*X. The regression slope ...
- 1,246
1
vote
How do we know the distribution of regression coefficients
The Central Limit Theorem gives, as is correctly stated in the question, the asymptotic normal distribution of the suitably standardised sample mean. This statement can not only be used to say ...
- 19.4k
3
votes
How do we know the distribution of regression coefficients
The estimated regression coefficients are functions of the data (both the response and explanatory variables). The regression model specifies the conditional distribution of the response given the ...
- 116k
3
votes
When failing to reject a null hypothesis can we also calculate the probability of observing the treatment assuming the alternative is true?
Yes we can, and in fact, this calculation is just a variation on the power function. Suppose you have a test using the parameter $\theta \in \Theta$ and with significance level $\alpha$. Then the ...
- 116k
0
votes
Anova with a specific reference group? Dunnett's test overall p-value?
I say the overall test you want is the usual F-test. You say you do not care if two treatment groups have different means. However, if two treatment groups have different means, then at least one of ...
- 55.1k
0
votes
Anova with a specific reference group? Dunnett's test overall p-value?
For your second question, yes, it is possible and it illustrates the issues with relying to "sig." vs. "not sig." This is not something to rely on. But don't take my word for it, ...
- 104k
4
votes
Why do we adjust for within-group variability in statistical testing of differences in group means?
Stephen gives a very good answer (+1). I wrote an entirely non-mathematical answer on Medium. Why analyze variances to compare means
which may appeal to the less mathematical among your students. (...
- 104k
5
votes
Accepted
Why do we adjust for within-group variability in statistical testing of differences in group means?
The issue is that you will always have differences between the responses observed in your two groups, simply because of random variations. Importantly, you will have such differences even if your ...
- 115k
4
votes
Accepted
Is a power analysis appropriate when the predictor variable is bounded between 0 and 1?
Yes, power analysis is absolutely valid in this case. It will work for any kind of predictor. After all, most predictors are bounded somehow by nature, and some can only take the values 0 and 1 (e.g., ...
- 115k
0
votes
Am I understanding critical value and p-value correctly?
An alternative is to forget about p-values and critical values. Instead, compute the difference between the two means, and the confidence interval for that difference. Then interpret the confidence ...
- 19k
0
votes
A specific example of two-sided chi-squared test
An old question, but for completeness:
Based on my understanding of your question, you're simply specifying an $\alpha=0.10$ error rate, and promoting that if one specifies this error rate (through a ...
- 334
2
votes
Practical significance, especially with percents: "standard" measure and threshold
To add to most other answers, as you were looking for references, below you'll find some saying that you should avoid using arbitrary thresholds.
Note that there are various standardized effect sizes ...
- 2,386
5
votes
Infinite upper confidence interval using Fisher exact test in R
Alternatively, I have created a bootstrap distribution for the odds ratio to get the confidence intervals (using the infer package in R), but I am not sure this is a valid approach.
Bootstrapping is ...
- 68.3k
16
votes
Infinite upper confidence interval using Fisher exact test in R
There's a one-sided confidence interval because you asked for a one-sided test
- 33k
2
votes
Detecting biased coins
The question asks us to distinguish between the possibilities $p = 0.5$ (unbiased) and $p = 0.57$ (biased) ("confirm... if the coin is actually biased or not, with a particular bias of p=0.57&...
- 37.1k
0
votes
Detecting biased coins
You can never confirm the bias is exactly 0.57, but you can do enough flips to rule out other values.
The confidence interval for the binomial proportion is, at its widest,
$$ \hat{p} \pm z_{1-\alpha/...
- 32.2k
6
votes
Accepted
Hypotheses Testing - Correlation vs. Regression
My responses are slightly different from Peter's, but I feel they are nonetheless relevant.
Do I need to run a correlation test before the linear regression?
As he said, you do not need to, but you ...
- 6,021
8
votes
Hypotheses Testing - Correlation vs. Regression
Welcome to CV.
For question A: No, you don't need to run correlation before regression, but you do need to check the assumptions of the model. Most of these checks are done after running the model.
...
- 104k
2
votes
If I have a very small n for one group and a very large number of features, should I choose a parametric or a non-parametric test?
In general, if the assumptions are met, parametric tests are more powerful than their non-parametric equivalents.
But the problem here is not parametric vs. non-parametric, it's sample size and power ...
- 104k
1
vote
Accepted
Test if a correlation is bigger than a specified value?
Using the Z transform its pretty easy to determine the sample size for this test. Suppose we want to test $H_0: r \leq \rho_1$, and we specify that for a correlation of $\rho_2$ ($\rho_1 < \rho_2$) ...
- 41
0
votes
Test if a correlation is bigger than a specified value?
The standard approach: Calculate a confidence interval for the true correlation $\rho$ and check if it excludes the hypothesized value $\rho_o$. In your case, if the lower 95 percent limit is higher ...
- 11.5k
2
votes
Accepted
Why is the sample standard deviation used in the z-test?
Generally, population parameters are unknown so we have to estimate them. If $\bar{x}$ is the sample mean, and $\hat{s}$ the sample standard deviation, then the statistic
$$ \dfrac{\bar{x}}{\hat{s} / ...
- 32.2k
0
votes
Computing a p-value of a 2-sided Chi-squared test for one variance
For a two tail test the p value has to be doubled. As the maximum value of p can be 1, we should take the value which after doubling will be less than 1. That is to say, right tail or left tail area ...
0
votes
Is there a statistical test for one participant measured many times?
Addendum to Original Question
I was at the beginning of my statistical knowledge when I first posed this question and since have been exposed to quite a variety of topics. Revisiting this question, I ...
- 6,021
2
votes
On the success or failure of an experiment
The problem is that you don't know what would have happened to the accident rate in the absence of the government campaign. It may have gone up, down, or remained relatively unchanged - lots of ...
- 17.2k
Top 50 recent answers are included