# Questions tagged [experiment-design]

The study of how to structure an information-gathering exercise where variation is present.

1,741 questions
Filter by
Sorted by
Tagged with
10 views

### How does one define what measure to use for a Regression Discontinuity design?

For example I am looking at the effectiveness of a reading learning program for 3rd graders. How do I define an acceptable cutoff measure? Do I look at accepted reading assessment tests, such as ...
19 views

### model.matrix correlations of discrete

I have the following full factorial design and I would like to evaluate the factors independency. ...
• 115
22 views

### Are these data pseudoreplicated?

I have an amalgamation of independently collected datasets covering a large area of ocean where surveys to count birds have occurred sporadically over a long time. Some surveys were from boat, some ...
• 123
1 vote
25 views

### When does a difference in means not capture the true treatment effects vs a regression with pre-treatment controls?

A question from Gelman - Regression & Other Stories... In answer to my own question...my understanding is that a difference in means should not capture the treatment effect when there are pre-...
11 views

### Blocking in $3^2$ fractional replicated design of experiments

I'm new to design of experiments and I'm planning an experiment which involves 2 factors at 3 levels ($3^2$ design). I'm using 3 levels as I think that there will be some non linearity, but I cannot ...
• 11
34 views

### How to generate a latin square for a user study?

I have to do a user study for a paper to see how using or not using tool X affects the quality of something (defined in my study). The study goes like this: participant 1 uses tool X to do task A and ...
18 views

### How to Analyze Latin Hypercube Results

After running an experiment generated by a Latin Hypercube design, what next? Just pick the best point? I guess there’s no ‘modeling’ or ‘curve fitting’ going on right? So you basically just choose ...
17 views

### Mixed ANOVA - Can We Standardize After Box-Cox Transformed Values

in our mixed ANOVA (2 timepoints, 3 interventiongroups, so 2x3 factorial ANOVA), the homogeneity of the residual variance was violated as indicated by significant levene tests, for the post-...
• 61
10 views

### AB Testing with Time Locked Metric of Interest

had a general AB test question surrounding how to best measure time locked performance of a metric of interest. The experiment basis is pretty simple: on a hosting website, users click on a button to ...
• 311
8 views

### Treating time as a continuous variable when analyzing a stepped wedge design

I am in the process of analyzing a stepped wedge clinical trial, examining the effect of an intervention aiming to improve rates of outcomes data collection (e.g. depression scales, quality of life ...
• 1,451
38 views

### Delta Method to calculate the standard error of ratio in an AB testing context

In AB testing context, if we have a control group and test group (2 groups), and I'd like to calculate the relative difference (Mean test/ Mean control -1) and the confidence interval of this ratio ...
18 views

### Right type of randomization / random allocation for low base rates

so I have a question on the correct "name" for the random-allocation to one of two treatment groups or one control group. We designed two interventions (readings material): one on ...
• 61
4 views

### How to address faulty randomization in RCBD design?

I have a client who wants me to analyze data from a randomized complete block design study. A large agricultural field was divided up into 16 plots and each row of plots was set up as a block ...
• 123
25 views

### Which hypothesis testing should I use for pageview/user during ab test?

I am currently running an experiment on a website, there are control and 1 variant. The variant changes one of the features of the website. Besides calculating the click-through-rate(CTR) of that ...
• 101
41 views

### Can a main effect coupled with an interaction, answer my question?

I am currently carrying out a research where I have two categorical predictors with two levels each. The research basically consists of presenting participants with one of four profiles of a target, ...
• 1
1 vote
36 views

### Why is SATE different to Treatment effect (difference in means)?

A question from Gelman - Regression & Other Stories. This one has me a bit stumped, I've reread sections of the chapter and I'm still not understanding why this is the case. I think the answer is ...
27 views

### special form of a paired design (very small, unbalaned)

I am trying to figure out how to analyze following design. Suppose three samples are taken from the starting material of an experiment which will be performed at larger scale (LS). The same ...
• 131
36 views

### Causal Inference for experiment

I'm working through a textbook (Regression and Other Stories) and have come across a particular problem that I am having difficulty convincing myself I understand. I am specifically interested in part ...
67 views

### Two way ANOVA unbalanced and nested TYPE III. Significant results

I have an unbalanced design with nested factors. In particular, the nested factor is obesity, with levels: of obese and nonobese. The other factor is group: women with syndrome, control women, and ...
1 vote
22 views

### If I'm conducting a matched study with multiple levels (e.g., students within classrooms), what is the appropriate level to match on?

Let's say that I'm conducting a non-randomized matching evaluating of an educational intervention to determine its impact on student grades. While the students receive the intervention, the teachers ...
• 157
21 views

### What is the suitable experimental design and how to implemante it in R or SPSS?

I have a question about the type of experimental design to describe my data and details of how to implement it using R or SPSS. I have three types of roses, three types of soil, two types of graft (...
1 vote
35 views

### Methods for evaluating a geo-experimental research design (DiD, CausalImpact or a third option?)

I'm doing my undergraduate thesis on an experimental design of the incrementality of clicks using Google Ads vs not using Google Ads. I've created a geo-experimental design, where I've made a split of ...
31 views

### Correlation coefficient of explanatory variables in D-optimal design

In the D-optimal design of design of experiments, some books make the following claims. The absolute value of the correlation coefficient becomes smaller I understand that the covariance is ...
38 views

### Blocking Binomial Data (Bernoulli Trials)

Imagine you have a colony of roaches and you want to compare the efficacies of two insecticides. On one day you apply insecticide A to 50 insects and record your "outcome" as "1" (...
21 views

### Nonparametric repeated measures with uneven number of subject repeats

I came across a dataset with some properties that make me uncertain what statistical tests would be appropriate for testing significance and effect size between certain conditions. Trying to keep the ...
• 1
9 views

### DoE for optimization / control approach?

I'm wondering whether a DoE approach could somehow be used as kind of an optimization algorithm? One of my current tasks is to find a set of five parameter which max a sixth one (see here for more: ...
• 2,187
1 vote
27 views

### AB Testing: Possible to use bootstrapping to obtain the probability that each variant is optimal?

I've recently thought of an extremely simple way to interpret an AB test. The only issue is, I'm not sure it's actually a valid approach — I do not have formal training in statistics, so this might be ...
1 vote
31 views

### AB Test, Experimental Design based on Machine Learning Output

Looking for the best way to set up an AB test. Scenario is that the online product I am working with has logged in users that pay a monthly fee, and they can upgrade to get access to more features. ...
28 views

### Calculating the power of an experiment

I'm working through a question in Gelman - Regression and Other Stories and I find myself getting confused with the following question; 'Power, type M error, and type S error: Consider the experiment ...
29 views

### How to properly (extrapolate?) information from samples to wider areas?

I have 12 plots, which I want to compare in terms of biomass production (the dry weight of the plants that grow there) in g per m2 and in kg per hectare. Imagine that 6 of these plots are full covered ...
54 views

### "Ordinal" is to "level of measurement" as "dependent variable" is to __________ [closed]

The Question If I'm drawing up a table that describes some variables -- a data dictionary, say -- and I consider whether a variable is "ordinal", or "nominal", or "interval&...
• 455
63 views

### What statistical test to use

I want to carry out a study but I don’t know which statistical method to use for analysing the results. I have two buildings (say, houses A and B) and I keep daily records of some quantitative ...
1 vote
32 views

### low sample size experimental design, t-test vs one way ANOVA

I'm designing a biological experiment wherein the sample size is very limited, as each sample is very hard to get. Moreover, I need to have 4 independent groups, each receiving a different treatment. ...
• 73
1 vote
31 views

### How to calculate how many levels of nuisance factor I need for a desired power level in my experiment

I have a nuisance factor that introduces a lot of variation in my data. I want to know how many levels of this factor I need to include in my blocked design to get a power level of, say, 90%. As an ...
1 vote
18 views

### Which item to administer to improve the standard error of the item difficulty estimate the most?

I have a student with a known ability level and a set of items with an estimated difficulty. The probability of a student getting an item correct follows an 1 parameter logistic model (also known as ...
• 6,126
1 vote
38 views

### a/b testing with two different treatment groups

I want to run an A/B test with two machine learning models M1 and M2 (called treatment groups ...
• 947
1 vote
30 views

### Can parallel designs be considered the "gold-standard" for RCTs

Statistically speaking, are parallel designs the "gold-standard" for RCTs? I mean statistically in opposition to financial, ethical or operational aspects. The only drawback I see for (...
1 vote
46 views

### 2x2 Repeated measures ANOVA gives DIFFERENT result in R and SPSS? (Same type of sums/data/analysis) Anyone know why?

Reposting this because I fixed the formatting to see if someone can actually provide a solution for this problem. I have a study design in which I am carrying out a 2x2 repeated measures ANOVA in ...
37 views

### CUPED - estimation of correlation with pre treatment covariates

I came across this variance reduction technique https://www.exp-platform.com/Documents/2013-02-CUPED-ImprovingSensitivityOfControlledExperiments.pdf the main idea is using correlation with a ...
• 305
31 views

### Small N analysis for a field experiment question, and rare events logit

so, I am running a "Field" experiment, meaning, people in a real life situation are subject to certain treatment, and I am trying to measure the effect of said treatment. In this case its an ...
• 3
60 views

### Calculate covariate-adjusted means and 95%CIs for treatment and control group separately for simple 2-arm trial

I have data for a simple 2-arm RCT, that looks as follows: ...
• 103
1 vote
55 views

### Why are global search algorithms not used in DoE?

I'm reading about Design of Experiments via various textbooks (e.g. Montgomery's), and powerful global optimization methods (such as Ant Colony Optimization) are not used. They rely only on a basic ...
19 views

### ATT vs ATE vs ITE

I was wondering what considerations should be made when choosing an appropriate estimator for treatment effect. E.g., suppose you have experimental data on email promotions, such that 1/3 of your ...
33 views

### Multiple Sclerosis: study design to beta test a software

I am a beta-tester of a software that is intended to help the radiologist to interpret MRI reading of Multiple Sclerosis (MS). MS is a disease that, over time, could lead to new lesions, expanding ...
• 133
16 views

### Does it make any sense to consider bootstrapping a $3^2$ factorial design?

Suppose I have two factors, $A = (A_1,A_2,A_3)$ and $B=(B_1,B_2,B_3)$, each at three levels, associated with a response $Y$, which is averaged over $N$ values. For example, the pairings would look ...
• 113
1 vote
40 views

### Is it better to have many measurements each with few data points or few measurements each with many data points?

I've wondered this for a long time. Let's say that I'm designing an experiment e.g. to measure human reaction time to images in which the contrast is varied. If I can afford 1000 trials I could make ...
• 111
19 views

### What statistical test to choose for study

I have an IV with 3 levels and then I am measuring anxiety scores before and after intervention. Is the before and after intervention a DV and is this a Mixed ANOVA?
1 vote
31 views

### Suggestion Needed about Design Experiment

As a part of my coursework, I'm supposed to be designing an experiment for an online cab service provider to understand the following: Whether letting the cab driver know the drop location of the ...
• 123