# Tag Info

Accepted

### When and how of stratification?

This is a big topic, but a few points for starters. Q1 "When is this necessary? (Aka Shouldn't pure randomization get "close enough"?)" Stratification is really only done for two ...
• 2,466

### CUPED vs. Diff-in-Diff

The comparison is apples to orangutans, but I will take a stab at it. CUPED is a variance reduction technique used in experimental analysis of in-test outcomes. It allows you to detect smaller effects ...
• 35.9k

### The rationale for when significance or null hypothesis testing is needed

I make my comments plus something an answer: There are situations in which it is clear from background knowledge that the null hypothesis is neither realistic nor interesting (such as "no ...
• 24.8k

### The rationale for when significance or null hypothesis testing is needed

This question is well answered by @ChristianHennig's comment, but that is worth some expansions. Statistical tests are useful when they allow some additional confidence in scientific inferences, but ...
• 15.7k

### The rationale for when significance or null hypothesis testing is needed

One scenario where this may be valid is when the $(1-\alpha)%$ confidence intervals of a statistic $\theta$ from 2 samples do not overlap; in such a scenario, a NHST will always be significant at the \$...
• 922
Accepted

### CUPED vs. Diff-in-Diff

i'm try to answer this question from a practitioner's perspective: in A/B testing, or online controlled experiment, when people mention CUPED, or DID, what are they actually doing and the assumptions ...
• 598
1 vote

### How to model a controlled experiment (three time points, two groups) in lme4?

The reason to not do the model where you treat time 0 as an observation is that you make additional assumptions about the distribution. These assumptions are sometimes badly violated, e.g. when you ...
• 32.8k
1 vote
Accepted

### How to model a controlled experiment (three time points, two groups) in lme4?

OK, I hope I understood correctly! First, I would perhaps forget T2 data altogether, if there was heavy drop out between T1 and T2. So, you can use the multilevel model approach or the covariate ...
• 2,233
1 vote

### How do I calculate the correlation of a variable between groups?

It's possible that you are being asked to calculate the correlation between the differences in average time between groups, probably relative to group 1. Each of these differences is a random number, ...
• 35.9k
1 vote
Accepted

### Power Analysis for Multiple Treatments per Respondent in Survey Experiment

You might be able to run the power calculations using the Superpower package for power calculations in ANOVAs (they also have a shiny app https://shiny.ieis.tue.nl/anova_power/) Another way would be ...
• 2,233
1 vote
Accepted

### How to compare initial means between weighted cases?

The outcome variable is not necessary in either cobalt or WeightIt. However, the estimand is. The estimand refers to which group ...
• 34.2k

Only top scored, non community-wiki answers of a minimum length are eligible