Skip to main content
6 votes
Accepted

When and how of stratification?

This is a big topic, but a few points for starters. Q1 "When is this necessary? (Aka Shouldn't pure randomization get "close enough"?)" Stratification is really only done for two ...
Alex J's user avatar
  • 2,466
5 votes

CUPED vs. Diff-in-Diff

The comparison is apples to orangutans, but I will take a stab at it. CUPED is a variance reduction technique used in experimental analysis of in-test outcomes. It allows you to detect smaller effects ...
dimitriy's user avatar
  • 35.9k
5 votes

The rationale for when significance or null hypothesis testing is needed

I make my comments plus something an answer: There are situations in which it is clear from background knowledge that the null hypothesis is neither realistic nor interesting (such as "no ...
Christian Hennig's user avatar
5 votes

The rationale for when significance or null hypothesis testing is needed

This question is well answered by @ChristianHennig's comment, but that is worth some expansions. Statistical tests are useful when they allow some additional confidence in scientific inferences, but ...
Michael Lew's user avatar
  • 15.7k
4 votes

The rationale for when significance or null hypothesis testing is needed

One scenario where this may be valid is when the $(1-\alpha)%$ confidence intervals of a statistic $\theta$ from 2 samples do not overlap; in such a scenario, a NHST will always be significant at the $...
jginestet's user avatar
  • 922
3 votes
Accepted

CUPED vs. Diff-in-Diff

i'm try to answer this question from a practitioner's perspective: in A/B testing, or online controlled experiment, when people mention CUPED, or DID, what are they actually doing and the assumptions ...
wei's user avatar
  • 598
1 vote

How to model a controlled experiment (three time points, two groups) in lme4?

The reason to not do the model where you treat time 0 as an observation is that you make additional assumptions about the distribution. These assumptions are sometimes badly violated, e.g. when you ...
Björn's user avatar
  • 32.8k
1 vote
Accepted

How to model a controlled experiment (three time points, two groups) in lme4?

OK, I hope I understood correctly! First, I would perhaps forget T2 data altogether, if there was heavy drop out between T1 and T2. So, you can use the multilevel model approach or the covariate ...
Sointu's user avatar
  • 2,233
1 vote

How do I calculate the correlation of a variable between groups?

It's possible that you are being asked to calculate the correlation between the differences in average time between groups, probably relative to group 1. Each of these differences is a random number, ...
dimitriy's user avatar
  • 35.9k
1 vote
Accepted

Power Analysis for Multiple Treatments per Respondent in Survey Experiment

You might be able to run the power calculations using the Superpower package for power calculations in ANOVAs (they also have a shiny app https://shiny.ieis.tue.nl/anova_power/) Another way would be ...
Sointu's user avatar
  • 2,233
1 vote
Accepted

How to compare initial means between weighted cases?

The outcome variable is not necessary in either cobalt or WeightIt. However, the estimand is. The estimand refers to which group ...
Noah's user avatar
  • 34.2k

Only top scored, non community-wiki answers of a minimum length are eligible