# Questions tagged [experiment-design]

The study of how to structure an information-gathering exercise where variation is present.

1,994 questions
Filter by
Sorted by
Tagged with
1 vote
34 views

### What's the main effect for a factor?

this is my first question on here. To add a little context: I'm studying the design of experiments and I'm having trouble with the part on the main effects of the factors. I understood that if you ...
33 views

### Sample size and test for 2×3 factorial design

I’m currently struggling with the design of my first survey. I’m working with a 2×3 factorial design where I’ve manipulated product-design stimuli based on two independent variables: brightness (dark,...
19 views

### Modelling intervention study using LME4 and participants are nested in subgroups in one group only

We conducted a behavior change field experiment using the following variables: Two time points (T0, T1) Two groups (intervention vs. control) Individual ID (n = 62 in group 1, n = 53 in group 2) ...
21 views

### Assessing effect of tutor in a small study without a control group

Suppose I have the following "experiment" setup. I am teaching a class of N students. All students were offered access to a TA. Each student gets K homework assignments throughout the year. ...
1 vote
35 views

### Discrepancy of a sample on Latin Hypercube Sampling

I'm currently working on Design of Experiments (DoE) in order to obtain data to, later on, make some FEM simulations. The starting point is: we want to make those simulations (that depends on some ...
• 111
28 views

### How to hypothesis test week-over-week metric for "significance", contrasting between test and control groups?

I have conducted an experiment to test a hypothesis that changing the layout of my website would increase overall user engagement with the website. I therefore assigned 50% of the website visitors to ...
• 207
46 views

### Understanding Ignorability and Confounding Variables

I am reading Data Analysis Using Regression and Hierarchical Models and am confused by the concept of ignorability. The description in the book seems to say different things. Said another way, we ...
• 123
24 views

### How should I test for parallel trend assumption in a repeated cross-sectional data?

I have a repeated cross-sectional dataset on time-use which I am using to estimate the impact of a certain shock. My outcome of interest is time spent by certain individual on particular activity. ...
39 views

### Selection of reasonable Power and Alpha for Sample Size calculation

I'm performing an A-B test which involves response rates to communications sent out to customers. I'm looking to calculate a required sample size, and I don't know the expected response rate of the ...
• 121
9 views

### How to combine intention-to-treatment randomization with block randomized experimental designs for within-differences in Power Analysis

I want to calculate the power for my experiment. Within multiple blocks of similarities (e.g., same gender, same pre-treatment outcome), respondents get the opportunity to take part in a treatment. As ...
20 views

### Interaction of endogenous variable with binary predictors (feols)

I would like to inquire about the correct way to specify interactions of endogenous variable with binary predictors (time and treat) with the feols function. Ideally I would like to specify a 3 way ...
64 views

### Experiment design to determine effect of ads on etsy shop performance

I am looking to design an experiment to determine the impact of turning on etsy ads for my shop. I opened a shop for plant pots and have made 27 sales in the 4 months I have been open. I am ...
• 41
20 views

### Dunnett's test for comparisons with Control - how to conduct ANOVA for 2 factors, as well as adjusting for different experimental designs?

From what I understand, before we conduct Dunnett's test for comparisons of treatments to a Control, we have to do ANOVA. However, for a 2-factor experiment, there seem to be a lot of problems to me. ...
• 53
13 views

### Experimental Design: Selecting value of $n$ given desired width of credible interval

Note that this is a cross post from here. I realize this is probably a better space Suppose I have $n$ IID Bernoulli trials with $k$ successes. Assume that as a prior we are assuming that $P(\theta)$ ...
• 113
18 views

### Design of experiment: non-constant factor

I want to set up a design of experiments (to study the growth rate of my cells) with 3 factors (3 levels), however one of my factors is hard to control, I can't keep it constant during the treatment/...
• 1
194 views

### How can restricted randomization to achieve covariate balance lead to imbalance in unobserved variables?

In literature, designing an experiment is considered a trade-off between covariate balance and robustness. For example Harshaw et al. (2024) writes In an effort to make the estimators more precise, ...
• 713
40 views

### Merging an administrative dataset with survey data in the context of a regression discontinuity design: what things should I consider?

I have an administrative dataset that contains the running variable for a Regression Discontinuity Design (RDD) study. I plan to merge this dataset with survey data that was collected for other ...
135 views

### Analyzing an experiment which consists of many "small" experiments

Consider the following setting: I am trying to choose between two bakers for my new bakery, Baker A and Baker B. The way that I am evaluating them is on a set of different cakes on the menu: Cake 1, ...
1 vote
32 views

### Logistic regression for experimental inference?

I'm curious if logistic regression is appropriate when the task is inferring a treatment effect on proportions (ex: success rate) when covariates are present and some imbalance across treatment and ...
• 3,374
27 views

### Bias in treatment effect estimation (adaptive design)

could someone explain what is the source of bias of treatment effect estimation in context of adaptive designs? The FDA guidance for industry for adaptive designs https://www.fda.gov/media/78495/...
• 195
34 views

### A/B Test for CTR without user-level data

Suppose I want to show two different ads and compare which one generates more clicks. By design I know that each user will see one of two ads at random every time s/he visits an external page where ...
• 21
38 views

### what happens at interim analysis?

I would like to understand what happens rather operationally during/after the interim analysis? if trial meets prespecified efficacy boundary, the study is unblinded at IA and the results announced ...
• 195
15 views

### Understanding Bayesian Optimal Experiment Design

I read this tutorial on Bayesian Experimentation Design (https://pyro.ai/examples/working_memory.html) and I'm trying to wrap my head around it. Suppose you have data (X,y). You're thinking about ...
148 views

### When and how of stratification?

I understand stratification at a novice level. For example, if we want to condition on gender in post experimentation inference, we might stratify or block on gender. As I understand it, we take each ...
• 3,374
1 vote
31 views

### Justifying uneven group sizes in an incomplete counterbalanced measures design using a balanced Latin square

I'm working on the statistical analysis of an experiment where we showed 6 videos to a group of people and asked them to rate their effectiveness. In order to handle bias due to the order, we divided ...
403 views

### The rationale for when significance or null hypothesis testing is needed

Why do people sometimes claim that an effect is so huge and "obvious" that it does not warrant any inferential statistics calculation, even though the sample size is not large? This is ...
• 175
1 vote
113 views

### How to model a controlled experiment (three time points, two groups) in lme4?

We conducted a behavior change field experiment using the following variables: Three time points (T0, T1, T2) Two groups (intervention vs. control) Individual ID Workshop ID The intervention was ...
19 views

### adaptive design based on non-comparative data

please help to understand the paragraph from FDA guidance on Adaptive Designs for Clinical Trials of Drugs and Biologics https://www.fda.gov/media/78495/download (Section IV) This section addresses ...
• 195
73 views

### How do I calculate the correlation of a variable between groups?

I have a question about calculating correlation in spss. I am part of a group in which we ran an experimental study in which there were three conditions (so we have three groups: 1, 2 and 3). One of ...
• 1
21 views

### Randomization of experiment runs to account for uncontrollable factors

I am running an experiment where there is a factor I suspect that can be affecting the results, I can't control nor absolutely measure ( ie. I can detect differences between specimens but can not ...
• 111
7 views

### How to run a DoE when multiple data points are collected from each leg?

I have a batch process with 4 parameters. I wish to run a DoE to understand how sensitive the output is to these parameters. I typically produce 15 samples with each setting. I use Minitab to create, ...
1 vote
92 views

### Power Analysis for Multiple Treatments per Respondent in Survey Experiment

I am running a vignette survey experiment with 6 total treatment options (5 treatments, 1 control), where each respondent randomly receives 2 of the treatment options and answers the same set of ...
• 65
19 views

### Multi-reader study design: split-plot or fully crossed?

I am a radiologist designing a study where 230 CT scans of cancer patients will be evaluated by 5 radiologists. There will be two sets of evaluations: one where radiologist is aided by an AI Computer-...
• 286
40 views

### How to compare initial means between weighted cases?

I want to check if the mean value of a certain personality trait differs between young adults in organization X and young adults outside of the organization, who are identical in terms of education ...
• 33
363 views

### CUPED vs. Diff-in-Diff

I've recently come across a method known as "Controlled-experiment Using Pre-Existing Data" (CUPED), which is a regression adjustments technique to create unbiased estimators in experiments. ...
• 3,374
1 vote
17 views

### How to deal with MNAR data based on failures in a trial?

Background I have a dataset in which participants had to perform a certain task within a time frame. The outcome of this task is not binary, but there was a criterion that should be reached within the ...
• 115
41 views

### Is naive mean estimator uniformly worse than HT (IPW) or Hajek estimators in survey sampling? If not, why is it less discussed in the literature?

Consider a toy example: we are interested in the average height of $n$ students $\bar{\tau}=\frac{1}{n}\sum_{i=1}^n\tau_i$, but for some reason, we can only access a random subset $S$ of it. Every ...
• 295
38 views

### Within subject experiments done by (some) Psychologists

Thanks in advance for any responses or resources you can link me to. I've had more of an Econ/Stats training and recently I've been working with some people who have a Psych background at work. They ...
18 views

### Relative importance of equal sample size vs equal sample density

I am conducting a vegetation study of the effects of prescribed fire. I have an experimental area that will be burned and a control area that will not be burned. The experimental area has three types ...
• 123
1 vote
72 views

### Which statistical approach is best for diverse conversion rates in an A/B test?

Our software startup builds chat bots for ecommerce websites. The chatbot talks to customers that open the chat bot, and has the goal of closing the sale with the store’s main product. We have about ...
• 111
54 views

### Understanding "Multiplication/ Group Operation" in Fractional Experiments

There's a group operation, at least of sorts, in Fractional Factorial design that I'm trying to understand. For definiteness, let's say we have 3 factors; A,B,C , at two levels each . Please critique ...
• 579
218 views

### Double Blind Studies & Confounders

I'm currently reading The Book of Why, and on pg 147-149 it talks about double blind RCT being very highly regarded in experiments, especially in the way it address confounders. In theory, I can ...
• 155
76 views

### Experimental design: What gives you more statistical power: Increasing number of sub samples or replicates?

Say I have an experiment with a binary response: 0 or 1. Within each experiment, I put X number of individuals in a group (X sub samples) and measure the proportion that has a response of 1 (e.g. 2/4)....
1 vote
80 views

### How to proceed when the observed effect is smaller than the Minimum Detectable Effect?

Given an A/B test with an already defined sample size and variance, a power analysis was conducted to estimate the Minimum Detectable Effect (MDE) of a proportion. After the experiment was concluded, ...
32 views

### What test of reliability for repeated measurements of the same scale?

I need to ask you for help on what method to use to calculate the reliability and validity of a scale that I measured repeatedly on the same subjects during an experiment. Let me explain: normally ...
• 71
30 views

### Formulating a linear mixed model with multiple random effects when one is not independent

I have a randomized complete block design. I have formulated linear mixed models using R, the package lme4, and the function lmer for all measurements except for one, where I am stuck on the random ...
1 vote
29 views

### Power analysis for Blocked/Stratified design?

I'm curious how power analysis can be conducted when the scientist is stratifying on one or more variables? I came across this approach but it went over my head. Ideally, the answer would not be "...
• 3,374
9 views

### Exogenous but non-random event for DiD

I have a quick question regarding whether the following event might pose any issues for my study. I am interested in exploring perceptions of climate change. In the early 1970s, former dictator Park ...
• 553