New answers tagged

1 vote

Determination of sample size vs. repeated measurements in experimental design

I want to compare the performance of two versions of a widget using a 2-sample comparison test. H_0 = both versions perform the same, H_1 = widget.b performs worse than widget.a. Your hypotheses ...
DrJerryTAO's user avatar
  • 1,425
4 votes
Accepted

Power of two-sample z-test

Your formula is for a one-sample test, that is for testing the null hypothesis that the mean of a population has a given value given a single sample from it. You can check the 'one sample' calculation ...
George Savva's user avatar
  • 2,044
5 votes
Accepted

How to calculate sample size from hazard ratio?

The key reference here is Schoenfeld (1983), who gives the following formula: $$ \frac{(z_{1-\beta}+z_{1-\alpha})^2}{pA\times pB\times\text{log}(\text{HR})^2} $$ This will give you the number of ...
PBulls's user avatar
  • 3,658
5 votes

Determination of sample size vs. repeated measurements in experimental design

Repeated testing of one widget will allow you to determine the magnitude of measurement variation (noise) and, maybe, if there are time or test-dependent changes in the performance. Those things are ...
Michael Lew's user avatar
  • 14.9k
2 votes

Should we favor family-wise error rate control or statistical power?

Type I errors (false positives) should be protected against where they are more costly than type II errors (false negatives). There are many context-dependent factors that need to go into ...
Michael Lew's user avatar
  • 14.9k
2 votes
Accepted

Calculate sample size in an experiment with interactions

This looks like a two-way ANOVA disguised as a three-way ANOVA. The critical things I see that you are testing are these two factors: The effect of stimulation. The effect of treatment. (Optional) ...
Shawn Hemelstrand's user avatar
3 votes

How can power and effect size help me interpret my A/B test results?

I'll take an unpopular but correct stance. If you want to 'interpret' or understand, or do anything else beyond an automatic decision that would be relevant to acceptance testing of widgets, you ...
Michael Lew's user avatar
  • 14.9k
2 votes

How can power and effect size help me interpret my A/B test results?

Surely, they can help me interpret the results of my test, one way or another? Power is something computed conditional on some assumption about the true effect size. If the effect size was at least ...
Demetri Pananos's user avatar
4 votes
Accepted

How can power and effect size help me interpret my A/B test results?

Power and effect size together tell you the probability that you'll correctly find a significant result, if the true effect really is as large as the effect size. You've correctly noted that the p-...
Nuclear Hoagie's user avatar
3 votes

How can power and effect size help me interpret my A/B test results?

You wrote I could arbitrarily increase or decrease both power and effect size (and keep contant n) No, this is incorrect, unless no one else will look at your analysis. It's true that if you set a ...
Peter Flom's user avatar
  • 117k
2 votes

What is the optimal method for distinguishing lack of power from non-significance in linear mixed models?

It is important understand why certain data points are labelled as outliers. Are they true anomalies, or do they represent a valid, albeit extreme, variation within the data? Investigating the nature ...
Lynchian's user avatar
  • 158
6 votes

What is the optimal method for distinguishing lack of power from non-significance in linear mixed models?

This is not possible in principle, because once you have collected your data, there is a one-to-one relationship between the p value and power. A p value greater than 0.05 tells you that your study ...
Stephan Kolassa's user avatar

Top 50 recent answers are included