Skip to main content
10 votes
Accepted

Related low p-values that do not meet statistically significant thresholds

Interesting question, let me break it down into a few distinct problems: Do multiple borderline significant results indicate overall significance? The answer to this one is simple. No. If anything, ...
Frans Rodenburg's user avatar
9 votes

Ideal test/p-value calculation for difference in means with small sample size and right skewed data?

Honestly, trying to do statistics on a sample of size 2 is like being "up the creek, w/o a paddle". This is not, and never will be, a representative sample of any population, and any ...
jginestet's user avatar
  • 1,849
8 votes

Ideal test/p-value calculation for difference in means with small sample size and right skewed data?

The limiting factor is the smaller of the two sample sizes. Because the power is exceptionally low in your situation, it is misleading to use p-values, as large p-values have no interpretations. If ...
Frank Harrell's user avatar
8 votes
Accepted

Variable is significant in multiple linear regression but not in t-test of the subgroups

You wrote: I am getting different p-values for a variable in t-test and its coefficient in multiple linear regression so I am unsure which one to believe. Believe them all, as long as the ...
Peter Flom's user avatar
  • 125k
7 votes

Ideal test/p-value calculation for difference in means with small sample size and right skewed data?

Ideal test/p-value calculation for difference in means with small sample size and right skewed data? There's really no "ideal". Normally you want two things: a significance level close to ...
Glen_b's user avatar
  • 286k
7 votes

getting insignificant p-values but large effect size what should I do?

You don't have a large effect size; you have a very uncertain effect size. It might be large. It might be small. It might be zero. It might be in the opposite direction to what you see in the data. ...
Thomas Lumley's user avatar
6 votes

Welch t-test p-values are poorly calibrated for $N=2$ samples

(Nice observation!) The Satterthwaite degrees-of-freedom calculation is only ever claimed to be an approximation, so what you observe is not totally surprising. The behavior seems to vanish as the ...
Dave's user avatar
  • 65k
5 votes

Variable is significant in multiple linear regression but not in t-test of the subgroups

This question has multiple issues Location of the intercept One issue is that the main terms can be significant or not depending on the place of the intercept. This relates to R: Why does type III ...
Sextus Empiricus's user avatar
4 votes

Are there alternative statistics to a p-value in NHST?

Your last statement talks about the probability of a hypothesis being true. That can be approached by Bayesian statistics. But that is a whole topic that I will not get into here (but is a good ...
Greg Snow's user avatar
  • 52.5k
4 votes

Are there alternative statistics to a p-value in NHST?

To put it briefly, this is now an operational question. Your observations are sound. In fact, nearly every statistician has (or should) consider the exact problem as you've described it. Closely tied ...
AdamO's user avatar
  • 63.7k
4 votes

Welch t-test p-values are poorly calibrated for $N=2$ samples

If you really want to do this (which might not necessarily be a good idea), there are solutions to the Behrens-Fisher problem that guarantee the type I error is at most the nominal (typically 5%) ...
Martin Modrák's user avatar
3 votes

Welch t-test p-values are poorly calibrated for $N=2$ samples

Question Is there a known calibration for Welch's t-test for N=2 samples per condition? You can't calibrate this problem because it depends on the unknown ratio of the unequal variances. See Behrens–...
Sextus Empiricus's user avatar
3 votes

Comparison of two test metrics

You need to precisely define "better". Once you do this, the solution is probably simple. If "better" means "more correlated with delta" then do the correlations. If &...
Peter Flom's user avatar
  • 125k
3 votes

Variable is significant in multiple linear regression but not in t-test of the subgroups

Looking at the results and the boxplots, dogs have clearly larger values than cats. Looking at the difference between tasks A and B, values for task B look lower for cats but slightly larger for dogs (...
Christian Hennig's user avatar
2 votes

Variable is significant in multiple linear regression but not in t-test of the subgroups

(a1 and m1 are the same model.) What you're observing is that multiple regression is a more powerful approach than separate $t$-...
Frans Rodenburg's user avatar
2 votes
Accepted

Testing for difference in trends of binary-parameter proportion/fraction versus continuous parameter between two (or more) different populations

Yes, you could do this by comparing the slopes of a spline for $M$, using a logistic regression model. Here is an example with some made up data with random trends: ...
Frans Rodenburg's user avatar
1 vote

Can I use a simpler model (small R²) to get more statistically relevant results?

Study statistical principles and use subject matter knowledge to pre-specify a model that makes sense, adjusting for things that should be adjusted for. Then don’t handle missing values in an ad hoc ...
Frank Harrell's user avatar
1 vote
Accepted

Test whether my new method's type I error rate correctly matches chosen alpha

Empirical validation of null-rejection methods tldr: in null-rejection testing, the empirical rate is typically different than the nominal rate (due to approximations), but it only matters if it is ...
Guillaume Dehaene's user avatar

Only top scored, non community-wiki answers of a minimum length are eligible