25 votes
Accepted

Why the p-value of t.test() is not statistically significant when mean values look really different

I agree with @pikachu that the standard deviations are too large compared with the difference between means for a t test to find a significant difference. Thank you for posting your data. It is always ...
BruceET's user avatar
  • 55.7k
17 votes
Accepted

Welch test seems to perform much worse than equal variance t-test

You are using a flawed metric for the performance of the test by focusing only on the power to detect a difference, but ignoring the possibility of falsely rejecting the null hypothesis. If you just ...
Erik's user avatar
  • 7,239
16 votes
Accepted

Compare the statistical significance of the difference between two polynomial regressions in R

...
Roland's user avatar
  • 6,486
14 votes

How to calculate the PDF of the 'difference' between two Beta distributions?

I know this is a bit of an old question but for what it's worth there is an established closed-form solution to this problem, found by Pham-Gia, Turkkan, and Eng in 1993. It's a piecewise solution ...
Adam Kern's user avatar
  • 241
14 votes

Welch test seems to perform much worse than equal variance t-test

This is a concern about the power of the Welch Test as compared to the textbook t test. Let's set up little simulation experiment and see if there are any discrepancies. The things that we can ...
Demetri Pananos's user avatar
12 votes
Accepted

Comparison of ranked lists

Summary I share my thoughts in Details section. I think they are useful in identifying what we really want to achieve. I think that the main problem here is that you haven't defined what a rank ...
caveman's user avatar
  • 2,701
11 votes
Accepted

Check for significant difference between numbers of sightings per cardinal direction

First off, methods such as ANOVA and Kruskal-Wallis pay no attention to the circular nature of data such as yours. It's not clear how you imagine applying either, but if you intend to regard direction ...
Nick Cox's user avatar
  • 55.2k
11 votes

Why the p-value of t.test() is not statistically significant when mean values look really different

When you consider the difference between means you have to use a different unit than the simple absolute difference. Take into account that you are measuring the difference in means produced by two ...
rapaio's user avatar
  • 6,924
9 votes
Accepted

MLM / HLM Equal sample sizes needed between groups?

I would not advise dropping the "extra" 50 participants. This will lead to a reduction in statistical power. While, in a perfect world, it is better to have equal sized groups (more ...
Robert Long's user avatar
  • 59.7k
8 votes

Logistic regression or T test?

This doesn't really answer the question but may still be of some interest. The standard assumption of a two sample $t$-test is that the conditional normal distribution of $X$ given a binary variable $...
Jarle Tufto's user avatar
  • 10.7k
8 votes

Compare the statistical significance of the difference between two polynomial regressions in R

@Ronald 's answer is the best and it's widely applicable to many similar problems (for example, is there a statistically significant difference between men and women in the relationship between weight ...
DeltaIV's user avatar
  • 17.7k
8 votes
Accepted

Hypothesis testing for difference in medians vs. median difference

Data. There are some minor discrepancies (maybe from rounding) in your data table. The table below is what I get from inputting your x1 and x2. These are the values I will use: ...
BruceET's user avatar
  • 55.7k
8 votes

Why do I not get a p-value and F value from ANOVA in R?

...
Thomas Bilach's user avatar
8 votes

Why the p-value of t.test() is not statistically significant when mean values look really different

With respect to plotting the data, I'd like to point the R package ggbeeswarm. In this case I think it is better than a boxplot or violin plot. The horizontal segments are the quantiles at 5%, 50%, 95%...
dariober's user avatar
  • 4,120
7 votes

Comparison of ranked lists

Warning: it's a great question and I don't know the answer, so this is really more of a "what I would do if I had to": In this problem there are lots of degrees of freedom and lots of comparisons one ...
Sven's user avatar
  • 1,121
7 votes

Measure of difference or distinctiveness that's comparable across both discrete and continuous variables

Consider a binary logistic regression model to predict the probability of being in that segment vs. not in that segment. Compute measures of relative explained variation for all the predictors, where ...
Frank Harrell's user avatar
7 votes

Are post-hoc adjusted analyses more valid than initial analyses?

Does the statistically insignificant value from the post-hoc adjusted analysis supercede the initial analysis of the primary outcome which isn't adjusted for confounders? What I assume you're trying ...
Demetri Pananos's user avatar
7 votes

Using a t-test to test effect size

I'm not understanding why you don't use Cohen's $d$ or Hedges $g$ in this situation. It is specifically designed for this purpose...to measure the magnitude of the effect rather than testing the null ...
Shawn Hemelstrand's user avatar
6 votes
Accepted

How to interpret Mann-Whitney's statistical significance if median is equal?

The Mann-Whitney is not a test of medians. At best, the Mann-Whitney test can only be claimed to a be a test of differences in mean-rank between two populations' pooled ranking. You can easily ...
AdamO's user avatar
  • 61.9k
6 votes
Accepted

ANOVA three group test is significant but the difference is small

The hypothesis test is doing exactly what it claims to be able to do: it is flagging to the investigator that there is an unusually high F-stat, too high for the null hypothesis to be believable. ...
Dave's user avatar
  • 60.9k
6 votes

Why do I not get a p-value and F value from ANOVA in R?

Your model has as many parameters as measurements. This will always make the residuals zero. As a consequence the anova function will not gonna compute a $p$-value (...
Sextus Empiricus's user avatar
6 votes

Why the p-value of t.test() is not statistically significant when mean values look really different

You have too high standard deviation (uncertainty) and with selected probability (I assume $\alpha = 0.05$ that is by default) the confidence intervals are overlapping, thus no statistically ...
pikachu's user avatar
  • 753
6 votes

Using a t-test to test effect size

If you get a very small p-value, it means that your sample size was probably adequate to detect the real effect size in the population. So you can already calculate an estimate of this effect size in ...
J-J-J's user avatar
  • 3,852
5 votes

Comparison of ranked lists

This sounds like the 'Willcoxon signed-rank test' (wikipedia link). Assuming that the values of your ranks are from the same set (ie [1, 25]) then this is a paired-...
danodonovan's user avatar
5 votes
Accepted

Comparing distributions of unequal sample sizes

The answer to your bolded question is no. (And you don't need to upsample or downsample anything.) If A and B are both random samples from their respective populations the sample cdf will converge to ...
Glen_b's user avatar
  • 281k
5 votes

Investigating differences between populations

Let's think the problem as follows. Say $X=(X_1,X_2,..X_n)$ and $Y$ is a binary variable standing for the population : $Y=0$ means first population, $Y=1$ means second population. The null hypothesis ...
Benoit Sanchez's user avatar
5 votes
Accepted

If one group's mean differs from zero but the other does not, can we conclude that the groups are different?

There is no contradiction, because The Difference Between "Significant" and "Not Significant" is not Itself Statistically Significant (Gelman & Stern, The American Statistician, 2006). Gelman ...
Stephan Kolassa's user avatar
5 votes
Accepted

How to estimate the confidence interval of the difference of means?

Let $\mu_X$ and $\mu_Y$ denote the means of $X$ and $Y$ respectively. If the goal is to obtain a confidence interval for $(\mu_X-\mu_Y)$, Jackknife and bootstrap would both work well, assuming your ...
dlnB's user avatar
  • 2,279
5 votes

Mixed models. Random slopes only, mean and group centering?

This all depends on the nature of your study. When you fit random intercepts, without random slopes, this assumes that each subject has the same response to the treatment, but each subject has a ...
Robert Long's user avatar
  • 59.7k
5 votes

How do I approach a linear mixed effects model with a 2-level group?

I get an output that only lists group A but not group B for some reason. Presumably, group B is the reference level, and since R uses contrast coding by default, B will be included in the intercept ...
Robert Long's user avatar
  • 59.7k

Only top scored, non community-wiki answers of a minimum length are eligible