New answers tagged

2 votes
Accepted

Building a bootstrapped distribution to test reliability of empirical data

This sounds more like a candidate for a permutation test. Relevant for the conclusions of the study is whether a decrease in this number took place across generations. So your hypothesis is that ...
user avatar
  • 113k
0 votes

Interpretation of p-value near alpha level

Frame challenge: Focus on the data you gathered and on weighing hypotheses against each other in metrics meaningful to your domain, not on the cult of the $p$-value. The cult of the $p$-value is a ...
user avatar
1 vote

P-value under 5% but power under 80%

If you observe a p-value equal to the alpha level, then you have roughly $50\%$ power for a true effect that equals the observed effect. Observations that are closer to the null hypothesis will not be ...
user avatar
1 vote

P-value under 5% but power under 80%

Possible situations abound. Here a few: Suppose you are testing $H_0: \mu = 50,$ against $H_a: \mu \ne 50$ at significance level $\alpha = 0.01 = 1\%,$ using $n = 10$ observations from a normal ...
user avatar
  • 49.9k
3 votes
Accepted

How to assess economic significance in a log-log OLS model?

Statistical and economic significance are not the same concept. Broadly speaking, statistical significance in a regression tells you if a variable has an effect on the outcome variable (depending on ...
user avatar
1 vote

Interpretation of p-value near alpha level

One consideration that is missing from the discussion is the problem when you have more than one test. As your p-value defining significance increases there is a concomitant increase in the chance ...
user avatar
6 votes

P-value under 5% but power under 80%

Power depends on the $\alpha$-level, which you set, not on the p-value (which you calculate). If you do a power calculation and determine that you have $80\%$ power to detect the difference of ...
user avatar
  • 31.2k
6 votes

Interpretation of p-value near alpha level

I will also add to Jeremy Miles' answer that makes a number of valid points. (I wrote initially that "I disagree with the claim that the reason that statistical significance testing doesn't seem ...
user avatar
2 votes

What is the impact of duplicate data on the variance of regression coefficient?

The coefficients themselves will no change. Imagine you perform the analysis on the first dataset, and plot the regression line with the datapoints around the regression line. Now what would happen if ...
user avatar
  • 570
4 votes

Interpretation of p-value near alpha level

I would add to the excellent answer of Jeremy Miles by saying that how you treat your p-values also strongly depends on what you want to do with them. They get a bad reputation (and rightfully so), ...
user avatar
1 vote

Permutation testing: Which variable should be shuffled?

With respect to a simple correlation between a single X and a single Y, you could shuffle either. That said, the p-value isn't "how often the resulting correlation coefficient exceeds the ...
user avatar
30 votes
Accepted

Interpretation of p-value near alpha level

There are two different approaches to interpreting statistical significance - the Fisher way, and the Neyman-Pearson way. We smush these together (into what Gerd Gigerenzer has called a 'bastardised ...
user avatar
  • 14.4k
3 votes
Accepted

Statistical test to check if data come from two different lines or just one

This is a standard application of an interaction term in a model. With x as continuous-valued predictors and y as your outcomes, let S represent which set a data ...
user avatar
  • 62.4k
3 votes
Accepted

Usefulness of KS tests and other similar distribution comparing tests

This is one of those ideas that starts out sounding great but winds up being less helpful than one might hope. For instance, just because KS (or a similar test) says that the feature has a different ...
user avatar
  • 31.2k
1 vote
Accepted

How many values should one validate to achieve a desired confidence?

There are a few ways to think about this. If you want to use the confidence interval approach, then you should also look at the "finite population correction" (Googling for that phrase ...
user avatar
  • 46.9k
0 votes

Is confounding a source of Type I errors?

First, let's talk about what you mean by "confounding". If you mean common causes of the treatment and outcome, then there is no confounding in randomized controlled trials. If you mean the ...
user avatar
  • 21.9k
1 vote

Is confounding a source of Type I errors?

Confounding can result in the identification of spurious, non-causal relationships that you don't care about, but I would not characterize these as Type I errors since there is a real statistical ...
user avatar
5 votes
Accepted

Unusally high F value ANOVA

With sample sizes as large as those you will almost always find that the statistical test gives very low p-values. Divide those SD values by the square root of the sample sizes to see the estimated SD ...
user avatar
  • 11.2k
0 votes

How to compare variables with data from multiple experiments?

You could treat each repetition of your experiment as a i.e. you probably have a randomised complete block design. If you have replicated the experiment in each repetition then you can estimate block ...
user avatar
  • 113
1 vote

Wilcoxon.test in R will not calculate exact distribution due to ties (scipy.stats.wilcoxon will)

I don't have enough reputation points to add this as a comment, so although not an answer, hope it does help. I've adjusted your code a bit for convenience: ...
user avatar
  • 11
1 vote
Accepted

How do I test that two samples don’t differ significantly by their demographics?

You can run tests to check that demographic variables are balanced. However, it's not recommended to do so for a randomized study. As you say, you have to trust the randomization. What's recommended ...
user avatar
  • 2,156
1 vote

Which statistics test should I use?

Comment: A suggestion has been made to do a chi-squared test on your $2 \times 6$ table of counts to see if the distributions for High and Low are different. That is one reasonable possibility among ...
user avatar
  • 49.9k
7 votes

How to report a P-value?

When is it appropriate to give the exact P-value, instead of writing e.g. P<0.05 (also in case of non-significant P-values)? As a general guideline you want to convey as much information about ...
user avatar
1 vote

How to report a P-value?

In practice, people report the exact p-value then declare whether the hypothesis test is statistically significant. Most statisticians don't advocate this. if the goal is hypothesis testing then ...
user avatar
  • 53k
0 votes

Statistical analysis (comparison) of time course experiments

Is there a tried-and-true statistical method to get a single p-value to determine whether two such curves are significantly different? Unless you have some theoretical form for the change in signal ...
user avatar
  • 62.4k
3 votes

Why do discrete choice models (such as MNL) not require test set?

Multinomial logistic regression can and often does consider out-of-sample performance. For instance, LeCun (1998) applied multinomial logistic regression to the pixels of the MNIST handwritten digits, ...
user avatar
  • 31.2k
1 vote

Is a p-value of 0.04993 enough to reject null hypothesis?

The 0.05 threshold is a hurdle that you have set for yourself in order to enforce a degree of self-skepticism about your alternative hypothesis. It somewhat weakens that self-skepticism if you change ...
user avatar
1 vote

Is a p-value of 0.04993 enough to reject null hypothesis?

In light of the assumptions of your model, you should reject the null because dichotomizing claims based on hypothesis tests have clear epistemological and pragmatic functions. But never forget that: ...
user avatar
0 votes

How to determine which variables are statistically significant in multiple regression?

Yes, you should look at the last column which contains the p-value parameter. Usually, we consider that if p-value < 0.05 for a certain variable then it is significant and has some relationship ...
user avatar
2 votes

Correlations significant at individual level, but not at group level?

See the answer at Correlation: average observations Vs observations: When you correlate averages, it will tend to be stronger because averaging reduces noise/variance. But, since the number of pens (...
user avatar
1 vote

How to calculate number of participants required to compare mean scores on a questionnaire between two groups?

It is a pretty old question here but since I know of, and made use of a straightforward way of calculating this I ‘d like to share it depending on whether you wish to have a double sided or a single ...
user avatar
  • 11
0 votes

References: t-test and Chi-squared test can be conducted with unequal sample sizes

Many intermediate-level applied statistical texts have the warnings mentioned in my comment. (One example, among many, is the text by Ott & Longnecker.) Use Welch, not pooled t test. Here is an ...
user avatar
  • 49.9k
0 votes

How can I compare robust linear mixed models and get the p-values, F statistic, AIC and BIC?

This answer comes late to the OP but I found this post trying to get the answer to the OP's question and then later found part of the answer. @Wagliss is correct that you cannot obtain an AIC or BIC ...
user avatar
1 vote

Collinearity problem

Probably yes, we can suppose that BMI and weight are highly correlated. You can verify it by performing a correlation between BMI and weight and checking the resulting coefficient, but as in previous ...
user avatar
  • 29
0 votes

Simple question on statistical comparison of grand/pooled means

I infer that your question is what can be done about this. You need to take advantage, carefully, of the numbers of relevant observations going into each "mean of means" and of the ...
user avatar
  • 62.4k
0 votes

Can I compare just some grades to see if they are statistically significantly different from one another?

I think this is a misunderstanding of what statistics does. You cannot in any meaningful sense do a test to state that student X's grade in one test is significantly different from student Y's. ...
user avatar
  • 11.9k
1 vote
Accepted

Is a huge F-statistic OK?

Let $n$ be the sample size. Define $RSS$ to be the residual sum of squares for the model, and define $TSS$ to be the total sum of squares. That is: $$ RSS = \sum_{i=1}^n (y_i - \hat y_i)^2\\ TSS = \...
user avatar
  • 31.2k
1 vote
Accepted

When to drop correlated features?

"Significant correlation" would usually mean that you tested a null hypothesis that $\rho=0$. Depending on your sample size, such correlation may still be quite close to zero. Why would you ...
user avatar
  • 113k
0 votes

difference in proportion in multiple sets

To be clear, your table suggests strategies, but the 'round' numbers would be unusual for real data. You'd need data consisting of counts of punches from boxing matches between boxer A and each of ...
user avatar
  • 49.9k
1 vote

Significance Testing of Difference in Variances of Autocorrelated Time Series Data

There's a problem here: Testing the difference in variances when? The time series you've shown are clearly non-stationary. What this means is that there is a pattern over time. More formally, a time ...
user avatar

Top 50 recent answers are included