Questions tagged [experiment-design]
The study of how to structure an information-gathering exercise where variation is present.
1,741
questions
0
votes
0
answers
10
views
How does one define what measure to use for a Regression Discontinuity design?
For example I am looking at the effectiveness of a reading learning program for 3rd graders.
How do I define an acceptable cutoff measure? Do I look at accepted reading assessment tests, such as ...
- 901
0
votes
0
answers
19
views
model.matrix correlations of discrete
I have the following full factorial design and I would like to evaluate the factors independency.
...
- 115
0
votes
1
answer
22
views
Are these data pseudoreplicated?
I have an amalgamation of independently collected datasets covering a large area of ocean where surveys to count birds have occurred sporadically over a long time. Some surveys were from boat, some ...
- 123
1
vote
1
answer
25
views
When does a difference in means not capture the true treatment effects vs a regression with pre-treatment controls?
A question from Gelman - Regression & Other Stories...
In answer to my own question...my understanding is that a difference in means should not capture the treatment effect when there are pre-...
- 87
0
votes
0
answers
11
views
Blocking in $3^2$ fractional replicated design of experiments
I'm new to design of experiments and I'm planning an experiment which involves 2 factors at 3 levels ($3^2$ design). I'm using 3 levels as I think that there will be some non linearity, but I cannot ...
- 11
0
votes
0
answers
34
views
How to generate a latin square for a user study?
I have to do a user study for a paper to see how using or not using tool X affects the quality of something (defined in my study). The study goes like this: participant 1 uses tool X to do task A and ...
0
votes
1
answer
18
views
How to Analyze Latin Hypercube Results
After running an experiment generated by a Latin Hypercube design, what next? Just pick the best point? I guess there’s no ‘modeling’ or ‘curve fitting’ going on right? So you basically just choose ...
0
votes
0
answers
17
views
Mixed ANOVA - Can We Standardize After Box-Cox Transformed Values
in our mixed ANOVA (2 timepoints, 3 interventiongroups, so 2x3 factorial ANOVA), the homogeneity of the residual variance was violated as indicated by significant levene tests,
for the post-...
- 61
0
votes
0
answers
10
views
AB Testing with Time Locked Metric of Interest
had a general AB test question surrounding how to best measure time locked performance of a metric of interest.
The experiment basis is pretty simple: on a hosting website, users click on a button to ...
- 311
0
votes
0
answers
8
views
Treating time as a continuous variable when analyzing a stepped wedge design
I am in the process of analyzing a stepped wedge clinical trial, examining the effect of an intervention aiming to improve rates of outcomes data collection (e.g. depression scales, quality of life ...
- 1,451
0
votes
2
answers
38
views
Delta Method to calculate the standard error of ratio in an AB testing context
In AB testing context, if we have a control group and test group (2 groups), and I'd like to calculate the relative difference (Mean test/ Mean control -1) and the confidence interval of this ratio ...
0
votes
0
answers
18
views
Right type of randomization / random allocation for low base rates
so I have a question on the correct "name" for the random-allocation to one of two treatment groups or one control group. We designed two interventions (readings material): one on ...
- 61
0
votes
0
answers
4
views
How to address faulty randomization in RCBD design?
I have a client who wants me to analyze data from a randomized complete block design study. A large agricultural field was divided up into 16 plots and each row of plots was set up as a block ...
- 123
0
votes
0
answers
25
views
Which hypothesis testing should I use for pageview/user during ab test?
I am currently running an experiment on a website, there are control and 1 variant. The variant changes one of the features of the website.
Besides calculating the click-through-rate(CTR) of that ...
- 101
0
votes
2
answers
41
views
Can a main effect coupled with an interaction, answer my question?
I am currently carrying out a research where I have two categorical predictors with two levels each. The research basically consists of presenting participants with one of four profiles of a target, ...
- 1
1
vote
1
answer
36
views
Why is SATE different to Treatment effect (difference in means)?
A question from Gelman - Regression & Other Stories. This one has me a bit stumped, I've reread sections of the chapter and I'm still not understanding why this is the case.
I think the answer is ...
- 87
0
votes
0
answers
27
views
special form of a paired design (very small, unbalaned)
I am trying to figure out how to analyze following design.
Suppose three samples are taken from the starting material of an experiment which will be performed at larger scale (LS). The same ...
- 131
0
votes
1
answer
36
views
Causal Inference for experiment
I'm working through a textbook (Regression and Other Stories) and have come across a particular problem that I am having difficulty convincing myself I understand.
I am specifically interested in part ...
- 87
0
votes
2
answers
67
views
Two way ANOVA unbalanced and nested TYPE III. Significant results
I have an unbalanced design with nested factors. In particular, the nested factor is obesity, with levels: of obese and nonobese. The other factor is group: women with syndrome, control women, and ...
1
vote
0
answers
22
views
If I'm conducting a matched study with multiple levels (e.g., students within classrooms), what is the appropriate level to match on?
Let's say that I'm conducting a non-randomized matching evaluating of an educational intervention to determine its impact on student grades. While the students receive the intervention, the teachers ...
- 157
0
votes
0
answers
21
views
What is the suitable experimental design and how to implemante it in R or SPSS?
I have a question about the type of experimental design to describe my data and details of how to implement it using R or SPSS.
I have three types of roses, three types of soil, two types of graft (...
- 11
1
vote
0
answers
35
views
Methods for evaluating a geo-experimental research design (DiD, CausalImpact or a third option?)
I'm doing my undergraduate thesis on an experimental design of the incrementality of clicks using Google Ads vs not using Google Ads.
I've created a geo-experimental design, where I've made a split of ...
- 11
0
votes
1
answer
31
views
Correlation coefficient of explanatory variables in D-optimal design
In the D-optimal design of design of experiments, some books make the following claims.
The absolute value of the correlation coefficient becomes smaller
I understand that the covariance is ...
user18616709
0
votes
1
answer
38
views
Blocking Binomial Data (Bernoulli Trials)
Imagine you have a colony of roaches and you want to compare the efficacies of two insecticides. On one day you apply insecticide A to 50 insects and record your "outcome" as "1" (...
- 105
0
votes
0
answers
21
views
Nonparametric repeated measures with uneven number of subject repeats
I came across a dataset with some properties that make me uncertain what statistical tests would be appropriate for testing significance and effect size between certain conditions.
Trying to keep the ...
- 1
0
votes
0
answers
9
views
DoE for optimization / control approach?
I'm wondering whether a DoE approach could somehow be used as kind of an optimization algorithm?
One of my current tasks is to find a set of five parameter which max a sixth one (see here for more: ...
- 2,187
1
vote
0
answers
27
views
AB Testing: Possible to use bootstrapping to obtain the probability that each variant is optimal?
I've recently thought of an extremely simple way to interpret an AB test. The only issue is, I'm not sure it's actually a valid approach — I do not have formal training in statistics, so this might be ...
1
vote
1
answer
31
views
AB Test, Experimental Design based on Machine Learning Output
Looking for the best way to set up an AB test.
Scenario is that the online product I am working with has logged in users that pay a monthly fee, and they can upgrade to get access to more features.
...
- 11
0
votes
0
answers
28
views
Calculating the power of an experiment
I'm working through a question in Gelman - Regression and Other Stories and I find myself getting confused with the following question;
'Power, type M error, and type S error: Consider the experiment ...
- 87
0
votes
0
answers
29
views
How to properly (extrapolate?) information from samples to wider areas?
I have 12 plots, which I want to compare in terms of biomass production (the dry weight of the plants that grow there) in g per m2 and in kg per hectare. Imagine that 6 of these plots are full covered ...
- 1
0
votes
1
answer
54
views
"Ordinal" is to "level of measurement" as "dependent variable" is to __________ [closed]
The Question
If I'm drawing up a table that describes some variables -- a data dictionary, say -- and I consider whether a variable is "ordinal", or "nominal", or "interval&...
- 455
0
votes
0
answers
63
views
What statistical test to use
I want to carry out a study but I don’t know which statistical method to use for analysing the results.
I have two buildings (say, houses A and B) and I keep daily records of some quantitative ...
1
vote
1
answer
32
views
low sample size experimental design, t-test vs one way ANOVA
I'm designing a biological experiment wherein the sample size is very limited, as each sample is very hard to get. Moreover, I need to have 4 independent groups, each receiving a different treatment.
...
- 73
1
vote
1
answer
31
views
How to calculate how many levels of nuisance factor I need for a desired power level in my experiment
I have a nuisance factor that introduces a lot of variation in my data. I want to know how many levels of this factor I need to include in my blocked design to get a power level of, say, 90%.
As an ...
- 11
1
vote
0
answers
18
views
Which item to administer to improve the standard error of the item difficulty estimate the most?
I have a student with a known ability level and a set of items with an estimated difficulty. The probability of a student getting an item correct follows an 1 parameter logistic model (also known as ...
- 6,126
1
vote
1
answer
38
views
a/b testing with two different treatment groups
I want to run an A/B test with two machine learning models M1 and M2 (called treatment groups ...
- 947
1
vote
0
answers
30
views
Can parallel designs be considered the "gold-standard" for RCTs
Statistically speaking, are parallel designs the "gold-standard" for RCTs?
I mean statistically in opposition to financial, ethical or operational aspects. The only drawback I see for (...
- 89
1
vote
0
answers
46
views
2x2 Repeated measures ANOVA gives DIFFERENT result in R and SPSS? (Same type of sums/data/analysis) Anyone know why?
Reposting this because I fixed the formatting to see if someone can actually provide a solution for this problem. I have a study design in which I am carrying out a 2x2 repeated measures ANOVA in ...
- 36
0
votes
0
answers
37
views
CUPED - estimation of correlation with pre treatment covariates
I came across this variance reduction technique https://www.exp-platform.com/Documents/2013-02-CUPED-ImprovingSensitivityOfControlledExperiments.pdf
the main idea is using correlation with a ...
- 305
0
votes
0
answers
31
views
Small N analysis for a field experiment question, and rare events logit
so, I am running a "Field" experiment, meaning, people in a real life situation are subject to certain treatment, and I am trying to measure the effect of said treatment.
In this case its an ...
- 3
0
votes
1
answer
60
views
Calculate covariate-adjusted means and 95%CIs for treatment and control group separately for simple 2-arm trial
I have data for a simple 2-arm RCT, that looks as follows:
...
- 103
1
vote
1
answer
55
views
Why are global search algorithms not used in DoE?
I'm reading about Design of Experiments via various textbooks (e.g. Montgomery's), and powerful global optimization methods (such as Ant Colony Optimization) are not used. They rely only on a basic ...
- 13
0
votes
0
answers
19
views
ATT vs ATE vs ITE
I was wondering what considerations should be made when choosing an appropriate estimator for treatment effect.
E.g., suppose you have experimental data on email promotions, such that 1/3 of your ...
- 21
0
votes
1
answer
33
views
Multiple Sclerosis: study design to beta test a software
I am a beta-tester of a software that is intended to help the radiologist to interpret MRI reading of Multiple Sclerosis (MS).
MS is a disease that, over time, could lead to new lesions, expanding ...
- 133
0
votes
0
answers
16
views
Does it make any sense to consider bootstrapping a $3^2$ factorial design?
Suppose I have two factors, $A = (A_1,A_2,A_3)$ and $B=(B_1,B_2,B_3)$, each at three levels, associated with a response $Y$, which is averaged over $N$ values. For example, the pairings would look ...
- 113
1
vote
0
answers
40
views
Is it better to have many measurements each with few data points or few measurements each with many data points?
I've wondered this for a long time. Let's say that I'm designing an experiment e.g. to measure human reaction time to images in which the contrast is varied.
If I can afford 1000 trials I could make ...
- 111
0
votes
0
answers
19
views
What statistical test to choose for study
I have an IV with 3 levels and then I am measuring anxiety scores before and after intervention. Is the before and after intervention a DV and is this a Mixed ANOVA?
- 1
1
vote
0
answers
31
views
Suggestion Needed about Design Experiment
As a part of my coursework, I'm supposed to be designing an experiment for an online cab service provider to understand the following:
Whether letting the cab driver know the drop location of the ...
- 123
0
votes
1
answer
38
views
How to determine power using standard error and effect size?
I was reading this article published from airbnb around gaurdrail metrics. In it, they discuss this power guardrail they derived but it's very unclear how they went about it.
How are they determining ...
- 1
1
vote
1
answer
42
views
Is there a difference between ignorability and strong ignorability?
The whole idea around ignorability is still leaving me a bit confused. I did read this post from the site: Strong ignorability: confusion on the relationship between outcomes and treatment. It had ...
- 266