# Tag Info

Accepted

### Building a bootstrapped distribution to test reliability of empirical data

This sounds more like a candidate for a permutation test. Relevant for the conclusions of the study is whether a decrease in this number took place across generations. So your hypothesis is that ...

### Interpretation of p-value near alpha level

Frame challenge: Focus on the data you gathered and on weighing hypotheses against each other in metrics meaningful to your domain, not on the cult of the $p$-value. The cult of the $p$-value is a ...
1 vote

### P-value under 5% but power under 80%

If you observe a p-value equal to the alpha level, then you have roughly $50\%$ power for a true effect that equals the observed effect. Observations that are closer to the null hypothesis will not be ...
1 vote

### P-value under 5% but power under 80%

Possible situations abound. Here a few: Suppose you are testing $H_0: \mu = 50,$ against $H_a: \mu \ne 50$ at significance level $\alpha = 0.01 = 1\%,$ using $n = 10$ observations from a normal ...
Accepted

### How to assess economic significance in a log-log OLS model?

Statistical and economic significance are not the same concept. Broadly speaking, statistical significance in a regression tells you if a variable has an effect on the outcome variable (depending on ...
1 vote

### Interpretation of p-value near alpha level

One consideration that is missing from the discussion is the problem when you have more than one test. As your p-value defining significance increases there is a concomitant increase in the chance ...

### P-value under 5% but power under 80%

Power depends on the $\alpha$-level, which you set, not on the p-value (which you calculate). If you do a power calculation and determine that you have $80\%$ power to detect the difference of ...

### Interpretation of p-value near alpha level

I will also add to Jeremy Miles' answer that makes a number of valid points. (I wrote initially that "I disagree with the claim that the reason that statistical significance testing doesn't seem ...

### What is the impact of duplicate data on the variance of regression coefficient?

The coefficients themselves will no change. Imagine you perform the analysis on the first dataset, and plot the regression line with the datapoints around the regression line. Now what would happen if ...

### Interpretation of p-value near alpha level

I would add to the excellent answer of Jeremy Miles by saying that how you treat your p-values also strongly depends on what you want to do with them. They get a bad reputation (and rightfully so), ...
1 vote

### Permutation testing: Which variable should be shuffled?

With respect to a simple correlation between a single X and a single Y, you could shuffle either. That said, the p-value isn't "how often the resulting correlation coefficient exceeds the ...
Accepted

### Interpretation of p-value near alpha level

There are two different approaches to interpreting statistical significance - the Fisher way, and the Neyman-Pearson way. We smush these together (into what Gerd Gigerenzer has called a 'bastardised ...
Accepted

### Statistical test to check if data come from two different lines or just one

This is a standard application of an interaction term in a model. With x as continuous-valued predictors and y as your outcomes, let S represent which set a data ...
Accepted

### Usefulness of KS tests and other similar distribution comparing tests

This is one of those ideas that starts out sounding great but winds up being less helpful than one might hope. For instance, just because KS (or a similar test) says that the feature has a different ...
1 vote
Accepted

### How many values should one validate to achieve a desired confidence?

There are a few ways to think about this. If you want to use the confidence interval approach, then you should also look at the "finite population correction" (Googling for that phrase ...

### Is confounding a source of Type I errors?

First, let's talk about what you mean by "confounding". If you mean common causes of the treatment and outcome, then there is no confounding in randomized controlled trials. If you mean the ...
1 vote

### Is confounding a source of Type I errors?

Confounding can result in the identification of spurious, non-causal relationships that you don't care about, but I would not characterize these as Type I errors since there is a real statistical ...
Accepted

### Unusally high F value ANOVA

With sample sizes as large as those you will almost always find that the statistical test gives very low p-values. Divide those SD values by the square root of the sample sizes to see the estimated SD ...

### How to compare variables with data from multiple experiments?

You could treat each repetition of your experiment as a i.e. you probably have a randomised complete block design. If you have replicated the experiment in each repetition then you can estimate block ...
1 vote

### Wilcoxon.test in R will not calculate exact distribution due to ties (scipy.stats.wilcoxon will)

I don't have enough reputation points to add this as a comment, so although not an answer, hope it does help. I've adjusted your code a bit for convenience: ...
1 vote
Accepted

### How do I test that two samples don’t differ significantly by their demographics?

You can run tests to check that demographic variables are balanced. However, it's not recommended to do so for a randomized study. As you say, you have to trust the randomization. What's recommended ...
1 vote

### Which statistics test should I use?

Comment: A suggestion has been made to do a chi-squared test on your $2 \times 6$ table of counts to see if the distributions for High and Low are different. That is one reasonable possibility among ...

### How to report a P-value?

When is it appropriate to give the exact P-value, instead of writing e.g. P<0.05 (also in case of non-significant P-values)? As a general guideline you want to convey as much information about ...
1 vote

### How to report a P-value?

In practice, people report the exact p-value then declare whether the hypothesis test is statistically significant. Most statisticians don't advocate this. if the goal is hypothesis testing then ...

### Statistical analysis (comparison) of time course experiments

Is there a tried-and-true statistical method to get a single p-value to determine whether two such curves are significantly different? Unless you have some theoretical form for the change in signal ...

### Why do discrete choice models (such as MNL) not require test set?

Multinomial logistic regression can and often does consider out-of-sample performance. For instance, LeCun (1998) applied multinomial logistic regression to the pixels of the MNIST handwritten digits, ...
1 vote

### Is a p-value of 0.04993 enough to reject null hypothesis?

The 0.05 threshold is a hurdle that you have set for yourself in order to enforce a degree of self-skepticism about your alternative hypothesis. It somewhat weakens that self-skepticism if you change ...
1 vote

### Is a p-value of 0.04993 enough to reject null hypothesis?

In light of the assumptions of your model, you should reject the null because dichotomizing claims based on hypothesis tests have clear epistemological and pragmatic functions. But never forget that: ...

### How to determine which variables are statistically significant in multiple regression?

Yes, you should look at the last column which contains the p-value parameter. Usually, we consider that if p-value < 0.05 for a certain variable then it is significant and has some relationship ...

### Correlations significant at individual level, but not at group level?

See the answer at Correlation: average observations Vs observations: When you correlate averages, it will tend to be stronger because averaging reduces noise/variance. But, since the number of pens (...
1 vote

### How to calculate number of participants required to compare mean scores on a questionnaire between two groups?

It is a pretty old question here but since I know of, and made use of a straightforward way of calculating this I ‘d like to share it depending on whether you wish to have a double sided or a single ...

### References: t-test and Chi-squared test can be conducted with unequal sample sizes

Many intermediate-level applied statistical texts have the warnings mentioned in my comment. (One example, among many, is the text by Ott & Longnecker.) Use Welch, not pooled t test. Here is an ...

### How can I compare robust linear mixed models and get the p-values, F statistic, AIC and BIC?

This answer comes late to the OP but I found this post trying to get the answer to the OP's question and then later found part of the answer. @Wagliss is correct that you cannot obtain an AIC or BIC ...
1 vote

### Collinearity problem

Probably yes, we can suppose that BMI and weight are highly correlated. You can verify it by performing a correlation between BMI and weight and checking the resulting coefficient, but as in previous ...

### Simple question on statistical comparison of grand/pooled means

I infer that your question is what can be done about this. You need to take advantage, carefully, of the numbers of relevant observations going into each "mean of means" and of the ...

### Can I compare just some grades to see if they are statistically significantly different from one another?

I think this is a misunderstanding of what statistics does. You cannot in any meaningful sense do a test to state that student X's grade in one test is significantly different from student Y's. ...
1 vote
Accepted

### Is a huge F-statistic OK?

Let $n$ be the sample size. Define $RSS$ to be the residual sum of squares for the model, and define $TSS$ to be the total sum of squares. That is:  RSS = \sum_{i=1}^n (y_i - \hat y_i)^2\\ TSS = \...
1 vote
Accepted

### When to drop correlated features?

"Significant correlation" would usually mean that you tested a null hypothesis that $\rho=0$. Depending on your sample size, such correlation may still be quite close to zero. Why would you ...