Questions tagged [experiment-design]
The study of how to structure an information-gathering exercise where variation is present.
1,861
questions
0
votes
0
answers
4
views
Create the same consistant optimized DoE with optFederov() [closed]
When I create a full factorial plan with the fac.design() function from the DoE.base package and I need to reduce the number of necessary runs, the ...
- 1
0
votes
0
answers
5
views
Inter-rater agreement for multiple factors for multiple pictures?
As part of my Master's dissertation, I am using photos and narratives to elicit a specific emotion. I have had 12 photos rated on various aspects of the emotion e.g. valence, arousal etc. I need ...
0
votes
0
answers
8
views
Analyzing data from a mixed experiment (nested or crossed?)
I am trying to analyze data from an experiment where my collaborators and I were interested in whether tweets about risk can influence people's risk perception (a continuous dependent variable). We ...
2
votes
0
answers
28
views
Draw Maximum Information from Experts to set up Utility Function
We aim to draw maximum information from CEP-logistics experts to set up a utility function for delivery chain choices for each parcel. The choices / dependent variables are specific transport chains, ...
- 31
0
votes
0
answers
34
views
+50
Are fractional factorial designs a part of 'Response Surface Methodology' or something different?
Fractional factorial designs (FFDs) and Response Surface Methodology (RSM) are both approaches to extracting some information about how multiple interacting factors affect some response variable. ...
- 15k
3
votes
0
answers
61
views
+50
Practical power analysis on linear model
Power analysis for linear model test asks for an analytical expression of the power of a linear model. The answer claims that
Note that this depends on the correlation between $X$ and $T$, $r_{TX}$.
...
- 209
0
votes
0
answers
7
views
Experiment terminology: control participants invertedly receive treatment
I have spent a solid hour to remember with the technical term for this. Suppose you have a between-subjects lab design with one control and two treatment conditions. What is the term to describe when ...
0
votes
0
answers
18
views
Split-split-plot design: Analysis and error terms
Can I have three levels of randomization and three error terms even when the replications are produced under same whole plot treatments under split split plot design? For example, three reps are ...
- 11
0
votes
0
answers
21
views
Do I need to use Latin Square when I only have one variable with two levels?
I am currently designing an acceptability judgment experiment that contains one variable with two levels. The experiment will be a within-subject design. There will be multiple sets of target ...
- 101
0
votes
0
answers
24
views
Is there a function in R to give the lack of fit and pure error of an ANOVA completed on a half 2^k factorial with centre points?
I have a half fraction factorial experiment for four factors, meaning I have eight experiments that I conduct. I expect that some of the responses are quadratic and not linear, thus I included centre ...
1
vote
1
answer
28
views
A/B test with two metrics, we test one metrics for improvement on the condition the other one doesn't change
I have the following business request:
"Test this change on the landing page and implement it for all the visitors if the test shows improvement in the conversion rate, but if the page load speed ...
- 131
7
votes
4
answers
754
views
Why is the Central Limit applicable in A/B testing?
I am having trouble understanding why the Central Limit Theorem (CLT) is applicable in A/B testing. As a beginner in statistics, I am trying to grasp the intuition behind it.
The CLT states that as we ...
- 71
0
votes
0
answers
11
views
Best DV operationalization for statistical power
I am setting up a questionnaire for a lab experiment to measure support for four competing policies. Very importantly, I want to know the public's order of preference. I am torn on how best to ...
0
votes
0
answers
19
views
Comparing treatment effects across groups
I created two donation campaigns, one loss framed (N:1993) and one gained framed (N:1989) I understood donations are usually veeeery low and the sample was a lot smaller than previously expected, so ...
- 3
2
votes
2
answers
46
views
Design of a Dental RCT in Which Patients Can Act as Their Own Control
I've been asked to help determine the design of an RCT. The design is primarily interested in the benefit of using a laser on some dental procedure in addition to a local anti-biotic and scaling, ...
- 30.7k
0
votes
0
answers
16
views
Permutation design for repeated measures via how() in RDA
We would like to properly set up the permutation design in anova() testing for our rda().
Study design
We chose several areas, ...
- 1
0
votes
0
answers
20
views
What's the null hypothesis of A/B test following a sample size calculation?
I have some confusion around what the null hypothesis should be for an A/B test. When we decide to do an A/B test and proceed to perform a sample size calculation. Let's say we work with the following ...
- 23
1
vote
0
answers
11
views
Does this experimental design avoid pseudo-replication?
I’m looking at a data set where 15 properties undertaking new management practices are sampled over three seasons. Measurements taken include variables such as vegetation cover and canopy cover. ...
0
votes
1
answer
40
views
A/B test - sample size calculation and unit of measurement
Does the unit of measurement change how we calculate sample size?
For example, if we were to run an A/B test on ads with our target metric being click-through-rate, and we are doing a sample size ...
- 51
0
votes
1
answer
36
views
A/B Testing Proportions - baseline rate is close to 1
I need a gut check here...
I'm trying to run an A/B test where the baseline metric is a conversion rate that is close to 1 - somewhere in the 80-85% range.
And so when I run my sample size calculation ...
- 51
0
votes
0
answers
13
views
Can we do some causal conclusions if the mediator follows the dependent variable?
We have a survey consisting of two waves ($N=4000$).
In Wave 1, we ask block of questions $M_1$.
In Wave 2, we ask blocks of questions $W$, $Y$, $M_2$
The order of questions is as follows:
$W \...
0
votes
1
answer
60
views
How to determine sample size for non-normal distributed data with continuous variables?
TLDR: I need to determine the minimum sample size for measuring the energy consumption of code.
Hello, I'm trying to determine how many samples (N) I need in an experiment to be confident that my ...
0
votes
1
answer
47
views
A/B test: measure likelihood of outcomes based on historical events
I am researching if it makes sense to create personalized emails based on past purchase behavior, but I do not know how to design A/B tests. The starting data I have is a list of about 500,000 ...
- 125
0
votes
0
answers
23
views
Random effects structure for logistic regression for RCBD with subsamples in glmer
I have a dataset that consists of a randomized complete block design with 4 blocks and 4 treatments. The response is a binary response of survival from approximately 80 subsamples per treatment. My ...
- 31
1
vote
1
answer
23
views
Experimental design with two factorial treatment and blocks of the treatment combination
I am going to do an experiment with planting seeds involving 2 factorial treatments. There are 4 families of seeds: F1, F2, F3, F4 and 3 foliar seed treatments: T1, T2, T3. There will be 160 seeds for ...
0
votes
0
answers
31
views
What is a principle of least perturbations?
Background
In BPF Performance Tools by Brendan Gregg, there is a section describing a series of 18 experiments measuring the overhead of tracing with eBPF on Linux.
In the description of the ...
1
vote
0
answers
15
views
How to combine within-subject score for between analysis when there is no within ID
I was task with analyzing some within/between-subject data, but my current guess is that I have no way of analyzing a combined within-subject score since an identifier/ID is missing. Is there a ...
- 11
1
vote
0
answers
74
views
Sample Size for categorical variables with given accuracy
I am trying to do a design of experiment to detect a defect (with little information given to me). In order to do that, I need to calculate the sample size. I am not really sure whether my attempt is ...
- 11
2
votes
2
answers
199
views
Model Sum of Squares from ANOVA table
Background: I am studying a course on statistical experiments using the textbook by Douglas Montgomery on the analysis of experiments. This is an introductory course and so I am relatively new to the ...
- 173
0
votes
0
answers
26
views
T-Test for Research Project (High School Student)?
I am completing a research project on how Heart Rate is affected by wearing two different types of shoes during 4 minute trials at VO2 Max (Reaching close to Max HR). One shoe is the Nike Vaporfly ...
1
vote
0
answers
32
views
Multilevel Bayesian model for Experimental Design with levels in treatment and control groups
Context
I'm designing an experiment and Bayesian analysis to interpret results. I've used Statistical Rethinking as inspiration for model structure.
Suppose I want to study the effects of a sports ...
- 2,123
1
vote
1
answer
70
views
Least Squares Estimation for CRD experiments
Setup
Let us use the following model for a Completely Randomised Designed (CRD) experiment:
$$ Y_{ij}= \mu + \tau_i + \epsilon_{ij}$$
where the errors $\epsilon _{ij}$ ~ Normal ($0, \sigma ^2 $) and ...
- 173
1
vote
1
answer
52
views
Understanding the advantages of CRD experiments
I am self studying an introductory course on Designed Experiments and have come across the notion of a Completely Randomised Design (CRD) defined as follows:
Completely Randomised Design: the ...
- 173
0
votes
0
answers
22
views
Difference-in-Difference: What to do with control observations when there are multiple possible intervention periods
I have several years worth of individual level data on several thousand individuals and a corresponding measure. During those years several hundred individuals underwent an intervention with the hope ...
6
votes
1
answer
371
views
Interpretation of causal effect from instrumental variables
We ran an experiment in which users could optionally watch a video. The video is intended to drive some other metric, $y$.
Because the intervention (video) was optional, choosing to watch the video ...
- 30.7k
0
votes
0
answers
16
views
BACI analysis where impact variable is continuous
I am analysing data from a before-after control-impact (BACI) experiment where all 10 sites (replicates) were measured once before a disturbance, then 5 sites were subjected to the disturbance, and ...
2
votes
0
answers
88
views
stepAIC in mixed models
As I haven't found the equivelant of the MASS::stepAIC for mixed models (eg in lmer) what I'm intending to do is to find the best lm model using stepAIC and then go in lmer and add the random effects.
...
- 295
5
votes
1
answer
140
views
Experimental design for a 96 well plate
I am designing an experiment that will include an assay on a 96 well plate and could use some advice on the optimal layout.
In our experiment we have 12 biological replicates, and each replicate has ...
- 1,192
2
votes
1
answer
93
views
What are balanced and orthogonal designed experiments?
I have recently started studying Designed Experiments and have have come across non-rigorous definitions of "balanced" and "orthogonal" experiments and would be interested in ...
- 173
0
votes
1
answer
130
views
Relationship between blocks, factors and treatments
I have recently began studying a course on Designed Experiments and am having some trouble understanding some of the terminology. I've looked at some other answers on the site and I think that I am ...
- 173
1
vote
0
answers
20
views
Surrogate model data requirements
I am trying to undersand how to approach the development of a surrogate model to investigate the associated uncertainty when predicting late arrival times. I.e there is an, to me, agnostic and ...
- 229
1
vote
0
answers
32
views
Between subject design, spread over multiple days
We have a between subjects design, testing for the impact of aesthetics on occupant satisfaction. Intervention is carpets vs no carpets.
The people participating in each leg (intervention vs control) ...
- 23
0
votes
1
answer
15
views
How much of a discrepency is tolerable if Sample Ratio Mismatch is detected?
We have analyzed several past online A/B tests and revealed that some of them have SRM at a statistically significant level (p<0.001).
Example: we had 50%/50% traffic allocation between control and ...
- 13
1
vote
0
answers
24
views
Treatment subspaces in Bailey's Design of Comparative Experiments
A draft of Bailey's Design of Comparative Experiments may be found here; I am referring to Chapter 2, figure 2.1.
Consider $t$ treatments. We have $N$ plots labeled $1, \dots, N$. We apply treatment $...
- 4,153
1
vote
0
answers
8
views
Estimating required sample size in cases when the base conversion rate always heavily fluctuates
I run marketing experiments on a website.
As far as I know, it's a well-known rule that we should estimate required sample size to detect a certain uplift in the treatment before running the ...
- 131
0
votes
0
answers
63
views
Calculating Sample Sizes for Tukey and Scheffe Pairwise Contrasts
$\newcommand{\x}[1]{\text{#1}}\newcommand{\op}[1]{\operatorname{#1}}$
This is Exercise 4.9 of Design and Analysis of Experiments, 2nd Ed., by Dean, Voss, and Draguljic.
Problem Statement: Suppose the ...
- 5,386
0
votes
0
answers
25
views
Clustering zip data at state level
Problem : Recomment 2 markets (States) for A/B test
Solution tried :
Cluster rows based simply by ZIP codes but then each state could appear across multiple clusters. How do I then dedup the results?...
1
vote
1
answer
53
views
Comparing total spent in unequal groups (lognormal distributed)
I have 100K customers from a population with a KPI name "spent" which is lognormal distributed.
The customers were split into 2 groups -
1st group - customers with an id ending with 0 - ...
- 11
2
votes
1
answer
38
views
How to decide time duration for synthetic control
For A/B test, one can use power analysis to determine the duration of experiment to run. What about synthetic control method? Is there a way to determine the duration of experiments post intervention?
- 63
1
vote
0
answers
33
views
Fractional factorial design with mixed categorical and numerical variables analysis for more than two levels
I have an experiment setup that consists of multiple continuous and multiple categorical variables. Right now, I am just using two levels for the categorical variables, allowing me to encode them as -...
- 111