Questions tagged [experiment-design]
The study of how to structure an information-gathering exercise where variation is present.
1,887
questions
0
votes
0
answers
25
views
What's the null hypothesis of A/B test following a sample size calculation?
I have some confusion around what the null hypothesis should be for an A/B test. When we decide to do an A/B test and proceed to perform a sample size calculation. Let's say we work with the following ...
- 23
1
vote
0
answers
12
views
Does this experimental design avoid pseudo-replication?
I’m looking at a data set where 15 properties undertaking new management practices are sampled over three seasons. Measurements taken include variables such as vegetation cover and canopy cover. ...
0
votes
1
answer
51
views
A/B test - sample size calculation and unit of measurement
Does the unit of measurement change how we calculate sample size?
For example, if we were to run an A/B test on ads with our target metric being click-through-rate, and we are doing a sample size ...
- 61
0
votes
1
answer
39
views
A/B Testing Proportions - baseline rate is close to 1
I need a gut check here...
I'm trying to run an A/B test where the baseline metric is a conversion rate that is close to 1 - somewhere in the 80-85% range.
And so when I run my sample size calculation ...
- 61
0
votes
0
answers
13
views
Can we do some causal conclusions if the mediator follows the dependent variable?
We have a survey consisting of two waves ($N=4000$).
In Wave 1, we ask block of questions $M_1$.
In Wave 2, we ask blocks of questions $W$, $Y$, $M_2$
The order of questions is as follows:
$W \...
0
votes
1
answer
68
views
How to determine sample size for non-normal distributed data with continuous variables?
TLDR: I need to determine the minimum sample size for measuring the energy consumption of code.
Hello, I'm trying to determine how many samples (N) I need in an experiment to be confident that my ...
0
votes
1
answer
51
views
A/B test: measure likelihood of outcomes based on historical events
I am researching if it makes sense to create personalized emails based on past purchase behavior, but I do not know how to design A/B tests. The starting data I have is a list of about 500,000 ...
- 125
0
votes
0
answers
32
views
Random effects structure for logistic regression for RCBD with subsamples in glmer
I have a dataset that consists of a randomized complete block design with 4 blocks and 4 treatments. The response is a binary response of survival from approximately 80 subsamples per treatment. My ...
- 31
1
vote
1
answer
29
views
Experimental design with two factorial treatment and blocks of the treatment combination
I am going to do an experiment with planting seeds involving 2 factorial treatments. There are 4 families of seeds: F1, F2, F3, F4 and 3 foliar seed treatments: T1, T2, T3. There will be 160 seeds for ...
0
votes
0
answers
31
views
What is a principle of least perturbations?
Background
In BPF Performance Tools by Brendan Gregg, there is a section describing a series of 18 experiments measuring the overhead of tracing with eBPF on Linux.
In the description of the ...
1
vote
0
answers
15
views
How to combine within-subject score for between analysis when there is no within ID
I was task with analyzing some within/between-subject data, but my current guess is that I have no way of analyzing a combined within-subject score since an identifier/ID is missing. Is there a ...
- 11
1
vote
0
answers
83
views
Sample Size for categorical variables with given accuracy
I am trying to do a design of experiment to detect a defect (with little information given to me). In order to do that, I need to calculate the sample size. I am not really sure whether my attempt is ...
- 11
2
votes
2
answers
263
views
Model Sum of Squares from ANOVA table
Background: I am studying a course on statistical experiments using the textbook by Douglas Montgomery on the analysis of experiments. This is an introductory course and so I am relatively new to the ...
- 213
0
votes
0
answers
28
views
T-Test for Research Project (High School Student)?
I am completing a research project on how Heart Rate is affected by wearing two different types of shoes during 4 minute trials at VO2 Max (Reaching close to Max HR). One shoe is the Nike Vaporfly ...
1
vote
0
answers
38
views
Multilevel Bayesian model for Experimental Design with levels in treatment and control groups
Context
I'm designing an experiment and Bayesian analysis to interpret results. I've used Statistical Rethinking as inspiration for model structure.
Suppose I want to study the effects of a sports ...
- 2,183
1
vote
1
answer
71
views
Least Squares Estimation for CRD experiments
Setup
Let us use the following model for a Completely Randomised Designed (CRD) experiment:
$$ Y_{ij}= \mu + \tau_i + \epsilon_{ij}$$
where the errors $\epsilon _{ij}$ ~ Normal ($0, \sigma ^2 $) and ...
- 213
1
vote
1
answer
70
views
Understanding the advantages of CRD experiments
I am self studying an introductory course on Designed Experiments and have come across the notion of a Completely Randomised Design (CRD) defined as follows:
Completely Randomised Design: the ...
- 213
0
votes
0
answers
24
views
Difference-in-Difference: What to do with control observations when there are multiple possible intervention periods
I have several years worth of individual level data on several thousand individuals and a corresponding measure. During those years several hundred individuals underwent an intervention with the hope ...
6
votes
1
answer
387
views
Interpretation of causal effect from instrumental variables
We ran an experiment in which users could optionally watch a video. The video is intended to drive some other metric, $y$.
Because the intervention (video) was optional, choosing to watch the video ...
- 31.3k
0
votes
0
answers
29
views
BACI analysis where impact variable is continuous
I am analysing data from a before-after control-impact (BACI) experiment where all 10 sites (replicates) were measured once before a disturbance, then 5 sites were subjected to the disturbance, and ...
2
votes
0
answers
157
views
stepAIC in mixed models
As I haven't found the equivelant of the MASS::stepAIC for mixed models (eg in lmer) what I'm intending to do is to find the best lm model using stepAIC and then go in lmer and add the random effects.
...
- 327
5
votes
1
answer
214
views
Experimental design for a 96 well plate
I am designing an experiment that will include an assay on a 96 well plate and could use some advice on the optimal layout.
In our experiment we have 12 biological replicates, and each replicate has ...
- 1,279
2
votes
1
answer
180
views
What are balanced and orthogonal designed experiments?
I have recently started studying Designed Experiments and have have come across non-rigorous definitions of "balanced" and "orthogonal" experiments and would be interested in ...
- 213
1
vote
1
answer
151
views
Relationship between blocks, factors and treatments
I have recently began studying a course on Designed Experiments and am having some trouble understanding some of the terminology. I've looked at some other answers on the site and I think that I am ...
- 213
1
vote
0
answers
20
views
Surrogate model data requirements
I am trying to undersand how to approach the development of a surrogate model to investigate the associated uncertainty when predicting late arrival times. I.e there is an, to me, agnostic and ...
- 238
1
vote
0
answers
32
views
Between subject design, spread over multiple days
We have a between subjects design, testing for the impact of aesthetics on occupant satisfaction. Intervention is carpets vs no carpets.
The people participating in each leg (intervention vs control) ...
- 23
0
votes
1
answer
29
views
How much of a discrepency is tolerable if Sample Ratio Mismatch is detected?
We have analyzed several past online A/B tests and revealed that some of them have SRM at a statistically significant level (p<0.001).
Example: we had 50%/50% traffic allocation between control and ...
- 13
1
vote
0
answers
28
views
Treatment subspaces in Bailey's Design of Comparative Experiments
A draft of Bailey's Design of Comparative Experiments may be found here; I am referring to Chapter 2, figure 2.1.
Consider $t$ treatments. We have $N$ plots labeled $1, \dots, N$. We apply treatment $...
- 4,203
1
vote
0
answers
12
views
Estimating required sample size in cases when the base conversion rate always heavily fluctuates
I run marketing experiments on a website.
As far as I know, it's a well-known rule that we should estimate required sample size to detect a certain uplift in the treatment before running the ...
- 131
0
votes
0
answers
71
views
Calculating Sample Sizes for Tukey and Scheffe Pairwise Contrasts
$\newcommand{\x}[1]{\text{#1}}\newcommand{\op}[1]{\operatorname{#1}}$
This is Exercise 4.9 of Design and Analysis of Experiments, 2nd Ed., by Dean, Voss, and Draguljic.
Problem Statement: Suppose the ...
- 5,615
0
votes
0
answers
26
views
Clustering zip data at state level
Problem : Recomment 2 markets (States) for A/B test
Solution tried :
Cluster rows based simply by ZIP codes but then each state could appear across multiple clusters. How do I then dedup the results?...
1
vote
1
answer
56
views
Comparing total spent in unequal groups (lognormal distributed)
I have 100K customers from a population with a KPI name "spent" which is lognormal distributed.
The customers were split into 2 groups -
1st group - customers with an id ending with 0 - ...
- 11
2
votes
1
answer
65
views
How to decide time duration for synthetic control
For A/B test, one can use power analysis to determine the duration of experiment to run. What about synthetic control method? Is there a way to determine the duration of experiments post intervention?
- 63
1
vote
0
answers
59
views
Fractional factorial design with mixed categorical and numerical variables analysis for more than two levels
I have an experiment setup that consists of multiple continuous and multiple categorical variables. Right now, I am just using two levels for the categorical variables, allowing me to encode them as -...
- 111
1
vote
0
answers
136
views
Split-plot design analysis by Linear Mixed Models in R
I am trying to analyse data from a Split-Plot experiment by Linear Mixed Models with lme4 in R. The design consists of a whole-plot factor and two split-plot factors (see attached layout). Whole-plot ...
- 21
3
votes
0
answers
23
views
Good text books for Design of Experiments [duplicate]
Could you please suggest some good textbooks on the subject of Design of Experiments particularly covering below topics:
Inference in linear models, One and two-way ...
- 353
1
vote
0
answers
76
views
How to analyze experimental data - 3x2 factorial design?
I am working on an experiment but am confused about what analyses I should undertake (I've picked the project up part way through). I’ve not been involved in experimental work in the past and so am ...
- 11
1
vote
1
answer
112
views
Advice for measuring MAUs (Monthly Active Users) through A/B testing?
thanks in advance for your advice.
I'm trying to determine whether certain product feature improvements increased the number of monthly active users (active is defined as whether the user made a ...
- 11
0
votes
0
answers
20
views
What’s a good scale robust similarity metric?
Context: While designing an experiment, there is interest from leadership in pairing control and test units together such that every (test, control) pair are as categorically similar as possible.
...
- 2,183
3
votes
1
answer
61
views
Should all the levels of a fixed factor be present in all the experimental trials?
I would like to analyze the output of 11 field trials dealing with the disturbance to soil after logging operations. The goal is to check if there are significant differences between two machinery: ...
2
votes
0
answers
36
views
Whats the model of this experiment
$36$ blood samples are collected at random from a blood bank and $6$ samples are sent to $6$ laboratories for studies. The assignment of each sample to each laboratory is random.
A friend of mine told ...
0
votes
0
answers
14
views
What intraclass correlation setup to use when there are a variable $k$ number of raters?
I have 200 subjects that need rating, each of which are given to between 3 to 8 raters (a rater may be able to grade more than one subject). I would like to compute the Intraclass Correlation but do ...
- 2,672
1
vote
0
answers
23
views
Can paired t-test be used when data comes from different task conditions at a single time point?
In my experiment a single group of participants performed a cognitive task with two interleaved types of blocks (A and B). Blocks A and B differed in terms of the type of stimuli used but were ...
- 11
0
votes
0
answers
17
views
Is there an alternatives to "one-tailed test" and "two-tailed test" in Bayesian A/B testing approach?
I am trying to grasp Bayesian A/B testing (so far I can't see it looks much different from standard Frequentist approach when using an uninformative prior).
A solution that we use for A/B testing ...
- 131
3
votes
1
answer
42
views
Comparing percentages!
Here is the problem I need help with,
I have two stem cell lines (line A and Line B). These cell lines were generated from one source.
I have six flasks, three of the flasks contain cells line A and ...
- 33
0
votes
0
answers
94
views
Regression Analysis for Stratified Random Sampling
I have some data that was found using a stratified random sampling procedure. We used this process to directly sample from the entire population and record Metrics that we were interested in. From my ...
- 23
0
votes
0
answers
26
views
Factorial study - test if there is a difference between mean level between the same factor
I am trying to answer the following question and I need a pointer.
The table below shows the lifetime of batteries made from 3 different types of material and at 3 different operating temperatures. ...
- 643
1
vote
1
answer
43
views
Experiment design - response variable is a sum over a period, low power
I would like to run a statistically rigorous experiment, similar to that of an E-commerce A/B test.
I want to create a checkerboard of time periods where I alternate between treatments A and B. ...
- 11
2
votes
1
answer
30
views
Design a model/statistical test to examine if college admission is disparate
Assuming the results (admitted and rejected) of all applicants are available, I'd like to design a regression or statistical test to examine if the admission is discriminative regarding the gender or ...
- 307
1
vote
1
answer
30
views
Blocking Design
Ran an experiment whereby I had a factor of 5 different levels on a plant cultivar. A wide range of responses were recorded. Due to the layout of the room, I quickly realised that blocking was ...
- 11