Questions tagged [experiment-design]

The study of how to structure an information-gathering exercise where variation is present.

Filter by
Sorted by
Tagged with
0 votes
0 answers
10 views

How does one define what measure to use for a Regression Discontinuity design?

For example I am looking at the effectiveness of a reading learning program for 3rd graders. How do I define an acceptable cutoff measure? Do I look at accepted reading assessment tests, such as ...
user avatar
0 votes
0 answers
19 views

model.matrix correlations of discrete

I have the following full factorial design and I would like to evaluate the factors independency. ...
user avatar
  • 115
0 votes
1 answer
22 views

Are these data pseudoreplicated?

I have an amalgamation of independently collected datasets covering a large area of ocean where surveys to count birds have occurred sporadically over a long time. Some surveys were from boat, some ...
user avatar
  • 123
1 vote
1 answer
25 views

When does a difference in means not capture the true treatment effects vs a regression with pre-treatment controls?

A question from Gelman - Regression & Other Stories... In answer to my own question...my understanding is that a difference in means should not capture the treatment effect when there are pre-...
user avatar
0 votes
0 answers
11 views

Blocking in $3^2$ fractional replicated design of experiments

I'm new to design of experiments and I'm planning an experiment which involves 2 factors at 3 levels ($3^2$ design). I'm using 3 levels as I think that there will be some non linearity, but I cannot ...
user avatar
  • 11
0 votes
0 answers
34 views

How to generate a latin square for a user study?

I have to do a user study for a paper to see how using or not using tool X affects the quality of something (defined in my study). The study goes like this: participant 1 uses tool X to do task A and ...
user avatar
0 votes
1 answer
18 views

How to Analyze Latin Hypercube Results

After running an experiment generated by a Latin Hypercube design, what next? Just pick the best point? I guess there’s no ‘modeling’ or ‘curve fitting’ going on right? So you basically just choose ...
user avatar
0 votes
0 answers
17 views

Mixed ANOVA - Can We Standardize After Box-Cox Transformed Values

in our mixed ANOVA (2 timepoints, 3 interventiongroups, so 2x3 factorial ANOVA), the homogeneity of the residual variance was violated as indicated by significant levene tests, for the post-...
user avatar
  • 61
0 votes
0 answers
10 views

AB Testing with Time Locked Metric of Interest

had a general AB test question surrounding how to best measure time locked performance of a metric of interest. The experiment basis is pretty simple: on a hosting website, users click on a button to ...
user avatar
  • 311
0 votes
0 answers
8 views

Treating time as a continuous variable when analyzing a stepped wedge design

I am in the process of analyzing a stepped wedge clinical trial, examining the effect of an intervention aiming to improve rates of outcomes data collection (e.g. depression scales, quality of life ...
user avatar
  • 1,451
0 votes
2 answers
38 views

Delta Method to calculate the standard error of ratio in an AB testing context

In AB testing context, if we have a control group and test group (2 groups), and I'd like to calculate the relative difference (Mean test/ Mean control -1) and the confidence interval of this ratio ...
user avatar
0 votes
0 answers
18 views

Right type of randomization / random allocation for low base rates

so I have a question on the correct "name" for the random-allocation to one of two treatment groups or one control group. We designed two interventions (readings material): one on ...
user avatar
  • 61
0 votes
0 answers
4 views

How to address faulty randomization in RCBD design?

I have a client who wants me to analyze data from a randomized complete block design study. A large agricultural field was divided up into 16 plots and each row of plots was set up as a block ...
user avatar
  • 123
0 votes
0 answers
25 views

Which hypothesis testing should I use for pageview/user during ab test?

I am currently running an experiment on a website, there are control and 1 variant. The variant changes one of the features of the website. Besides calculating the click-through-rate(CTR) of that ...
user avatar
0 votes
2 answers
41 views

Can a main effect coupled with an interaction, answer my question?

I am currently carrying out a research where I have two categorical predictors with two levels each. The research basically consists of presenting participants with one of four profiles of a target, ...
user avatar
  • 1
1 vote
1 answer
36 views

Why is SATE different to Treatment effect (difference in means)?

A question from Gelman - Regression & Other Stories. This one has me a bit stumped, I've reread sections of the chapter and I'm still not understanding why this is the case. I think the answer is ...
user avatar
0 votes
0 answers
27 views

special form of a paired design (very small, unbalaned)

I am trying to figure out how to analyze following design. Suppose three samples are taken from the starting material of an experiment which will be performed at larger scale (LS). The same ...
user avatar
  • 131
0 votes
1 answer
36 views

Causal Inference for experiment

I'm working through a textbook (Regression and Other Stories) and have come across a particular problem that I am having difficulty convincing myself I understand. I am specifically interested in part ...
user avatar
0 votes
2 answers
67 views

Two way ANOVA unbalanced and nested TYPE III. Significant results

I have an unbalanced design with nested factors. In particular, the nested factor is obesity, with levels: of obese and nonobese. The other factor is group: women with syndrome, control women, and ...
user avatar
1 vote
0 answers
22 views

If I'm conducting a matched study with multiple levels (e.g., students within classrooms), what is the appropriate level to match on?

Let's say that I'm conducting a non-randomized matching evaluating of an educational intervention to determine its impact on student grades. While the students receive the intervention, the teachers ...
user avatar
  • 157
0 votes
0 answers
21 views

What is the suitable experimental design and how to implemante it in R or SPSS?

I have a question about the type of experimental design to describe my data and details of how to implement it using R or SPSS. I have three types of roses, three types of soil, two types of graft (...
user avatar
1 vote
0 answers
35 views

Methods for evaluating a geo-experimental research design (DiD, CausalImpact or a third option?)

I'm doing my undergraduate thesis on an experimental design of the incrementality of clicks using Google Ads vs not using Google Ads. I've created a geo-experimental design, where I've made a split of ...
user avatar
0 votes
1 answer
31 views

Correlation coefficient of explanatory variables in D-optimal design

In the D-optimal design of design of experiments, some books make the following claims. The absolute value of the correlation coefficient becomes smaller I understand that the covariance is ...
user avatar
0 votes
1 answer
38 views

Blocking Binomial Data (Bernoulli Trials)

Imagine you have a colony of roaches and you want to compare the efficacies of two insecticides. On one day you apply insecticide A to 50 insects and record your "outcome" as "1" (...
user avatar
0 votes
0 answers
21 views

Nonparametric repeated measures with uneven number of subject repeats

I came across a dataset with some properties that make me uncertain what statistical tests would be appropriate for testing significance and effect size between certain conditions. Trying to keep the ...
user avatar
  • 1
0 votes
0 answers
9 views

DoE for optimization / control approach?

I'm wondering whether a DoE approach could somehow be used as kind of an optimization algorithm? One of my current tasks is to find a set of five parameter which max a sixth one (see here for more: ...
user avatar
  • 2,187
1 vote
0 answers
27 views

AB Testing: Possible to use bootstrapping to obtain the probability that each variant is optimal?

I've recently thought of an extremely simple way to interpret an AB test. The only issue is, I'm not sure it's actually a valid approach — I do not have formal training in statistics, so this might be ...
user avatar
1 vote
1 answer
31 views

AB Test, Experimental Design based on Machine Learning Output

Looking for the best way to set up an AB test. Scenario is that the online product I am working with has logged in users that pay a monthly fee, and they can upgrade to get access to more features. ...
user avatar
0 votes
0 answers
28 views

Calculating the power of an experiment

I'm working through a question in Gelman - Regression and Other Stories and I find myself getting confused with the following question; 'Power, type M error, and type S error: Consider the experiment ...
user avatar
0 votes
0 answers
29 views

How to properly (extrapolate?) information from samples to wider areas?

I have 12 plots, which I want to compare in terms of biomass production (the dry weight of the plants that grow there) in g per m2 and in kg per hectare. Imagine that 6 of these plots are full covered ...
user avatar
  • 1
0 votes
1 answer
54 views

"Ordinal" is to "level of measurement" as "dependent variable" is to __________ [closed]

The Question If I'm drawing up a table that describes some variables -- a data dictionary, say -- and I consider whether a variable is "ordinal", or "nominal", or "interval&...
user avatar
  • 455
0 votes
0 answers
63 views

What statistical test to use

I want to carry out a study but I don’t know which statistical method to use for analysing the results. I have two buildings (say, houses A and B) and I keep daily records of some quantitative ...
user avatar
1 vote
1 answer
32 views

low sample size experimental design, t-test vs one way ANOVA

I'm designing a biological experiment wherein the sample size is very limited, as each sample is very hard to get. Moreover, I need to have 4 independent groups, each receiving a different treatment. ...
user avatar
  • 73
1 vote
1 answer
31 views

How to calculate how many levels of nuisance factor I need for a desired power level in my experiment

I have a nuisance factor that introduces a lot of variation in my data. I want to know how many levels of this factor I need to include in my blocked design to get a power level of, say, 90%. As an ...
user avatar
1 vote
0 answers
18 views

Which item to administer to improve the standard error of the item difficulty estimate the most?

I have a student with a known ability level and a set of items with an estimated difficulty. The probability of a student getting an item correct follows an 1 parameter logistic model (also known as ...
user avatar
  • 6,126
1 vote
1 answer
38 views

a/b testing with two different treatment groups

I want to run an A/B test with two machine learning models M1 and M2 (called treatment groups ...
user avatar
1 vote
0 answers
30 views

Can parallel designs be considered the "gold-standard" for RCTs

Statistically speaking, are parallel designs the "gold-standard" for RCTs? I mean statistically in opposition to financial, ethical or operational aspects. The only drawback I see for (...
user avatar
1 vote
0 answers
46 views

2x2 Repeated measures ANOVA gives DIFFERENT result in R and SPSS? (Same type of sums/data/analysis) Anyone know why?

Reposting this because I fixed the formatting to see if someone can actually provide a solution for this problem. I have a study design in which I am carrying out a 2x2 repeated measures ANOVA in ...
user avatar
0 votes
0 answers
37 views

CUPED - estimation of correlation with pre treatment covariates

I came across this variance reduction technique https://www.exp-platform.com/Documents/2013-02-CUPED-ImprovingSensitivityOfControlledExperiments.pdf the main idea is using correlation with a ...
user avatar
  • 305
0 votes
0 answers
31 views

Small N analysis for a field experiment question, and rare events logit

so, I am running a "Field" experiment, meaning, people in a real life situation are subject to certain treatment, and I am trying to measure the effect of said treatment. In this case its an ...
user avatar
  • 3
0 votes
1 answer
60 views

Calculate covariate-adjusted means and 95%CIs for treatment and control group separately for simple 2-arm trial

I have data for a simple 2-arm RCT, that looks as follows: ...
user avatar
1 vote
1 answer
55 views

Why are global search algorithms not used in DoE?

I'm reading about Design of Experiments via various textbooks (e.g. Montgomery's), and powerful global optimization methods (such as Ant Colony Optimization) are not used. They rely only on a basic ...
user avatar
0 votes
0 answers
19 views

ATT vs ATE vs ITE

I was wondering what considerations should be made when choosing an appropriate estimator for treatment effect. E.g., suppose you have experimental data on email promotions, such that 1/3 of your ...
user avatar
0 votes
1 answer
33 views

Multiple Sclerosis: study design to beta test a software

I am a beta-tester of a software that is intended to help the radiologist to interpret MRI reading of Multiple Sclerosis (MS). MS is a disease that, over time, could lead to new lesions, expanding ...
user avatar
  • 133
0 votes
0 answers
16 views

Does it make any sense to consider bootstrapping a $3^2$ factorial design?

Suppose I have two factors, $A = (A_1,A_2,A_3)$ and $B=(B_1,B_2,B_3)$, each at three levels, associated with a response $Y$, which is averaged over $N$ values. For example, the pairings would look ...
user avatar
1 vote
0 answers
40 views

Is it better to have many measurements each with few data points or few measurements each with many data points?

I've wondered this for a long time. Let's say that I'm designing an experiment e.g. to measure human reaction time to images in which the contrast is varied. If I can afford 1000 trials I could make ...
user avatar
  • 111
0 votes
0 answers
19 views

What statistical test to choose for study

I have an IV with 3 levels and then I am measuring anxiety scores before and after intervention. Is the before and after intervention a DV and is this a Mixed ANOVA?
user avatar
  • 1
1 vote
0 answers
31 views

Suggestion Needed about Design Experiment

As a part of my coursework, I'm supposed to be designing an experiment for an online cab service provider to understand the following: Whether letting the cab driver know the drop location of the ...
user avatar
0 votes
1 answer
38 views

How to determine power using standard error and effect size?

I was reading this article published from airbnb around gaurdrail metrics. In it, they discuss this power guardrail they derived but it's very unclear how they went about it. How are they determining ...
user avatar
  • 1
1 vote
1 answer
42 views

Is there a difference between ignorability and strong ignorability?

The whole idea around ignorability is still leaving me a bit confused. I did read this post from the site: Strong ignorability: confusion on the relationship between outcomes and treatment. It had ...
user avatar

1
2 3 4 5
35