Questions tagged [experiment-design]
The study of how to structure an information-gathering exercise where variation is present.
1,887
questions
0
votes
0
answers
38
views
What method to find relation between variable A and B with unequal replicates in different plot
I have an experimental setting as illustrated below.
Two different set of soil variables (say A and B) were taken in multiple plots (Plot1, Plot2, Plot n, those are not properly plot maybe... rather ...
- 1
2
votes
0
answers
26
views
Sample size planning method differs from analysis method
I have seen some articles using one method for sample size estimation(two sample comparison for means or proportions) and subsequently using regression analysis/non parametric methods.
If I have a ...
- 1,335
0
votes
1
answer
35
views
Assessing the accuracy of a particular A/B testing design methodology
I commented to a colleague that we could improve our measurement of treatment effects by ensuring proper randomisation into treatment and control groups by balancing the treatment and control groups ...
- 49
0
votes
0
answers
42
views
Bayesian experiment analysis of multi-cell experiment
I have an experiment running with four cells divided by two axes; the axes are (1) experiment group (test, control) and demand period (high, low). There are four total combinations therein, test-high, ...
- 2,183
0
votes
0
answers
19
views
Why doesn't experimental design exhaust all possible cases
This is an experimental study of how clients how well individuals can assess the relative quality of financial advice and whether a catering strategy based on confirming a client’s opinions and ...
- 239
0
votes
0
answers
83
views
A/B testing conversion rate when a user can have multiple conversions (no longer Bernoulli)
How to perform a comparison of means test in a scenario where a user can have multiple conversions? In the case where a user can have at most 1 conversion, one can simply do a comparison of ...
- 401
0
votes
0
answers
186
views
difference between stratification and blocking in experimental design
What is the difference between stratification and blocking in experimental design(or randomized control trial)?
Blocking is to obtain homogeneity in a controlled factor. However, I can block more than ...
- 1,335
0
votes
0
answers
34
views
Sample size for a logistic regression [duplicate]
I would like to calculate the minimal number of participants (sample size) I need for my experimental protocol. I use logistic regression. I compare two groups in a protocol having three independent ...
- 13
1
vote
0
answers
47
views
Choice of statistical test
I would need some help in order to understand which statistical tests apply for my experimental design.
I want to compare two groups of participants in an experimental protocol in which the linguistic ...
- 13
0
votes
0
answers
50
views
ANOVA: crossed or nested
I wish to run a factorial design of experiment (DoE) to investigate the effect sizes of many factors. In my opinion the evaluation of the data depends on what I am comfortable to assume. If I am ...
- 701
3
votes
1
answer
79
views
Could I fit a third-order response surface given this experimental design?
My supervisors and I are planning to run an ecological experiment in the very near future. They have proposed an experimental design and I would like to get some feedback on it. In particular, I am ...
- 142
0
votes
0
answers
55
views
Statistical test across multiple subjects with multiple trials (under multiple conditions)
I have some behavioural data that I am trying to analyse. I am not sure about the correct statistical tests to run to answer my scientific questions.
I have a group of subjects. Each subject is ...
- 1
2
votes
0
answers
40
views
asking humans to rank items
I have around 50 items and I need to ask human graders to rank them. Is there any good resource on how to design this crowd sourcing task? For example, it will be tiresome to ask humans to rank 50 ...
- 121
5
votes
1
answer
159
views
is ab test valid for concluding causal relation, when analysis unit differs from randomization unit?
In a typical A/B test, the randomization unit is user level, sometimes the analysis unit may be page/visit level, like a cluster randomization experiment. In this situation, the iid assumption doesn't ...
- 301
1
vote
1
answer
44
views
R package for DoE with a single factor? [duplicate]
Is there a package for R available to construct an experiment design with only a single factor?
To be more precise, I want to evaluate the dependency of voltage on the temperature. The latter shall be ...
- 3,041
0
votes
0
answers
32
views
How to communicate comparing accuracy metrics between two different studies?
I'm working on a paper where we trained a statistical model and tested it on a dataset. We achieved some accuracy score.
There's another study that has a different accuracy score but in the same ...
1
vote
0
answers
28
views
Design of Experiment for one factor in R [closed]
Is there an package to design an experiment with a single factor, potentially with multiple levels?
I want to know how a certain voltage depends on temperature and for that I initially planned to ...
- 3,041
2
votes
0
answers
392
views
What are the assumptions of repeated measures ANCOVA?
In my experiment I have a randomly allocated control and experimental group, as well as pretest and posttest.
To control for baseline differences, I saw that conducting an ANCOVA is recommended. ...
- 21
0
votes
0
answers
16
views
Possibility of non-binary balanced incomplete block design
Balanced Incomplete Block Design (BIBD) is defined to be a block design which is proper, equireplicate, incomplete and binary such that every distinct pair of treatments appear together $\lambda$ ...
- 525
0
votes
1
answer
27
views
Help with a generalized linear mixed effects model of a forestry trial
I am having troubles coming up with the proper statistical model and R code for a forestry trial we did several years ago. We looked at whether wood piles left over in clear cuts were beneficial for ...
- 1
2
votes
1
answer
44
views
How does one define what measure to use for a Regression Discontinuity design?
For example I am looking at the effectiveness of a reading learning program for 3rd graders.
How do I define an acceptable cutoff measure? Do I look at accepted reading assessment tests, such as ...
- 1,008
0
votes
0
answers
48
views
model.matrix correlations of discrete
I have the following full factorial design and I would like to evaluate the factors independency.
...
- 327
2
votes
1
answer
65
views
Are these data pseudoreplicated?
I have an amalgamation of independently collected datasets covering a large area of ocean where surveys to count birds have occurred sporadically over a long time. Some surveys were from boat, some ...
- 144
1
vote
1
answer
52
views
When does a difference in means not capture the true treatment effects vs a regression with pre-treatment controls?
A question from Gelman - Regression & Other Stories...
In answer to my own question...my understanding is that a difference in means should not capture the treatment effect when there are pre-...
- 49
1
vote
0
answers
20
views
Blocking in $3^2$ fractional replicated design of experiments
I'm new to design of experiments and I'm planning an experiment which involves 2 factors at 3 levels ($3^2$ design). I'm using 3 levels as I think that there will be some non linearity, but I cannot ...
- 21
0
votes
0
answers
55
views
How to generate a latin square for a user study?
I have to do a user study for a paper to see how using or not using tool X affects the quality of something (defined in my study). The study goes like this: participant 1 uses tool X to do task A and ...
1
vote
1
answer
159
views
How to Analyze Latin Hypercube Results
After running an experiment generated by a Latin Hypercube design, what next? Just pick the best point? I guess there’s no ‘modeling’ or ‘curve fitting’ going on right? So you basically just choose ...
1
vote
2
answers
496
views
Delta Method to calculate the standard error of ratio in an AB testing context
In AB testing context, if we have a control group and test group (2 groups), and I'd like to calculate the relative difference (Mean test/ Mean control -1) and the confidence interval of this ratio ...
- 11
0
votes
0
answers
33
views
How to address faulty randomization in RCBD design?
I have a client who wants me to analyze data from a randomized complete block design study. A large agricultural field was divided up into 16 plots and each row of plots was set up as a block ...
- 205
0
votes
0
answers
45
views
Which hypothesis testing should I use for pageview/user during ab test?
I am currently running an experiment on a website, there are control and 1 variant. The variant changes one of the features of the website.
Besides calculating the click-through-rate(CTR) of that ...
- 101
0
votes
2
answers
50
views
Can a main effect coupled with an interaction, answer my question?
I am currently carrying out a research where I have two categorical predictors with two levels each. The research basically consists of presenting participants with one of four profiles of a target, ...
- 21
1
vote
1
answer
271
views
Why is SATE different to Treatment effect (difference in means)?
A question from Gelman - Regression & Other Stories. This one has me a bit stumped, I've reread sections of the chapter and I'm still not understanding why this is the case.
I think the answer is ...
- 49
0
votes
0
answers
29
views
special form of a paired design (very small, unbalaned)
I am trying to figure out how to analyze following design.
Suppose three samples are taken from the starting material of an experiment which will be performed at larger scale (LS). The same ...
- 131
0
votes
1
answer
178
views
Causal Inference for experiment
I'm working through a textbook (Regression and Other Stories) and have come across a particular problem that I am having difficulty convincing myself I understand.
I am specifically interested in part ...
- 49
0
votes
2
answers
442
views
Two way ANOVA unbalanced and nested TYPE III. Significant results
I have an unbalanced design with nested factors. In particular, the nested factor is obesity, with levels: of obese and nonobese. The other factor is group: women with syndrome, control women, and ...
1
vote
0
answers
24
views
If I'm conducting a matched study with multiple levels (e.g., students within classrooms), what is the appropriate level to match on?
Let's say that I'm conducting a non-randomized matching evaluating of an educational intervention to determine its impact on student grades. While the students receive the intervention, the teachers ...
- 201
1
vote
0
answers
142
views
Methods for evaluating a geo-experimental research design (DiD, CausalImpact or a third option?)
I'm doing my undergraduate thesis on an experimental design of the incrementality of clicks using Google Ads vs not using Google Ads.
I've created a geo-experimental design, where I've made a split of ...
- 11
0
votes
1
answer
45
views
Correlation coefficient of explanatory variables in D-optimal design
In the D-optimal design of design of experiments, some books make the following claims.
The absolute value of the correlation coefficient becomes smaller
I understand that the covariance is ...
user18616709
0
votes
1
answer
58
views
Blocking Binomial Data (Bernoulli Trials)
Imagine you have a colony of roaches and you want to compare the efficacies of two insecticides. On one day you apply insecticide A to 50 insects and record your "outcome" as "1" (...
- 105
1
vote
1
answer
27
views
DoE for optimization / control approach?
I'm wondering whether a DoE approach could somehow be used as kind of an optimization algorithm?
One of my current tasks is to find a set of five parameter which max a sixth one (see here for more: ...
- 3,041
1
vote
0
answers
225
views
AB Testing: Possible to use bootstrapping to obtain the probability that each variant is optimal?
I've recently thought of an extremely simple way to interpret an AB test. The only issue is, I'm not sure it's actually a valid approach — I do not have formal training in statistics, so this might be ...
1
vote
1
answer
40
views
AB Test, Experimental Design based on Machine Learning Output
Looking for the best way to set up an AB test.
Scenario is that the online product I am working with has logged in users that pay a monthly fee, and they can upgrade to get access to more features.
...
- 11
0
votes
0
answers
54
views
Calculating the power of an experiment
I'm working through a question in Gelman - Regression and Other Stories and I find myself getting confused with the following question;
'Power, type M error, and type S error: Consider the experiment ...
- 49
0
votes
1
answer
61
views
"Ordinal" is to "level of measurement" as "dependent variable" is to __________ [closed]
The Question
If I'm drawing up a table that describes some variables -- a data dictionary, say -- and I consider whether a variable is "ordinal", or "nominal", or "interval&...
- 679
0
votes
0
answers
68
views
What statistical test to use
I want to carry out a study but I don’t know which statistical method to use for analysing the results.
I have two buildings (say, houses A and B) and I keep daily records of some quantitative ...
1
vote
1
answer
99
views
low sample size experimental design, t-test vs one way ANOVA
I'm designing a biological experiment wherein the sample size is very limited, as each sample is very hard to get. Moreover, I need to have 4 independent groups, each receiving a different treatment.
...
- 73
1
vote
1
answer
38
views
How to calculate how many levels of nuisance factor I need for a desired power level in my experiment
I have a nuisance factor that introduces a lot of variation in my data. I want to know how many levels of this factor I need to include in my blocked design to get a power level of, say, 90%.
As an ...
- 11
2
votes
0
answers
24
views
Which item to administer to improve the standard error of the item difficulty estimate the most?
I have a student with a known ability level and a set of items with an estimated difficulty. The probability of a student getting an item correct follows an 1 parameter logistic model (also known as ...
- 7,181
2
votes
1
answer
890
views
a/b testing with two different treatment groups
I want to run an A/B test with two machine learning models M1 and M2 (called treatment groups ...
- 1,067
1
vote
0
answers
116
views
2x2 Repeated measures ANOVA gives DIFFERENT result in R and SPSS? (Same type of sums/data/analysis) Anyone know why?
Reposting this because I fixed the formatting to see if someone can actually provide a solution for this problem. I have a study design in which I am carrying out a 2x2 repeated measures ANOVA in ...
- 48