Questions tagged [experiment-design]
The study of how to structure an information-gathering exercise where variation is present.
1,975
questions
0
votes
0
answers
8
views
Exogenous but non-random event for DiD
I have a quick question regarding whether the following event might pose any issues for my study.
I am interested in exploring perceptions of climate change. In the early 1970s, former dictator Park ...
3
votes
0
answers
22
views
Regression models as goal metrics for experiments
What are the known issues or pitfalls associated with scoring regression/ML models on observations involved in a randomized experiment (a/b test, etc.)?
In a randomized experiment, for most metrics (...
0
votes
0
answers
6
views
expected number of events under H0 and under alternative [closed]
Imagine that I have a trial design with interim analysis. What is the expected number of events under the null hypothesis? Under the alternative? Could you please help me to understand what it is ...
0
votes
0
answers
14
views
Machine Learning Experiment: should training parameters be fixed for a valid comparisons across model?
I am training an autoencoder three times, each time on a different dataset. The three datasets all have the same number of features, but have vastly different sizes.
Assuming one of the datasets is A ...
0
votes
1
answer
44
views
Usefulness of Latin Square Design
I want to understand the usefulness of the Latin Square Design. Suppose there are two block factors, each with three blocks. There are three experimental groups: A, B, C. One realization of the Latin ...
0
votes
0
answers
10
views
Is there a best approach for statistical power for field experiment?
I'm setting up a field experiment and want to make a solid plan for the analysis, since that's whereI get stressed in the process. I will deploy 26 pairs of traps. 13 pairs will be in urban forest, ...
1
vote
0
answers
14
views
Is a permutation test valid in comparing a fraud block policy false positive rate across two different months?
I'm looking to conduct a counterfactual experiment in an e-commerce fraud context. When a fraud policy blocks a transaction you necessarily will never know if it was truly fraudulent. I'm trying to ...
0
votes
0
answers
17
views
How can I implement a linear mixed model on data where the reference doesn't have repeated measures?
I want to analyze in paired data that has many dependent variables (continuous) and several categorical predictors. The predictors are: Group (patient/control), Treatment (none, t1, t2, t3), and Gene (...
3
votes
1
answer
186
views
Reduce the model sequentially
I was given an ANOVA table and asked to reduce the model sequentially.
I searched the online resources say: When reducing the model sequentially, you typically start by assessing the significance of ...
2
votes
1
answer
44
views
Statistical Test or Regression for Intervention Effectiveness?
I'm looking to model the effectiveness of an intervention on human behavior. I have control groups and treatments groups for the outcome of interest and pre and post measurements (just 1 for each) for ...
1
vote
0
answers
18
views
Why can't interaction be measured with a complete block design?
I am struggling to understand why interaction cannot be measured in a CBD. While many sources say it is because there is only one observation per block/treatment combination, could you not compare the ...
0
votes
0
answers
32
views
Looking for an intuitive explanation of D-Criterion for Optimal Design Problem
I know only a little about Fisher information and optimal experimental design, but I'm trying to better understand the subject. If I have an experiment composed of a single detector and my detector ...
0
votes
1
answer
22
views
Show conditional independence in randomized experiment
I'm new to causal inference, and I have a question regarding a potential causal relationship. I'm wondering if the following assumption holds:
$(Y_i(1), Y_i(0), X_i) ⊥ Z_i -> (Y_i(1), Y_i(0)) ⊥ Z_i ...
2
votes
1
answer
37
views
Regression using unordered combinations, sign of predictor depends on order
First time poster here. Is there anything wrong with the following regression approach?
Background:
I am dealing with fuel consumption $F$ of a vehicle under a variety of conditions. I have developed ...
0
votes
0
answers
25
views
What is the correct way to analyze this data in a mixed model approach?
I have an experiment with rates of an waste in complete randomized blocks designs, with evaluation of soil fertility and tree heights/diameters over the years. Which is the best model to fit this data?...
7
votes
2
answers
582
views
average revenue per user increases each country but decreases when combined
I'm trying to figure out what could be some reasons when we do the AB testing we're seeing average revenue per user increases for each country but when we calculate the overall average revenue per ...
3
votes
1
answer
81
views
How is it reasonable that randomised controlled trials can be used to perform causal inference?
I understand that randomised controlled trials (RCTs) are used to perform causal inference, but I'm a confused about how this is reasonable. Let's say that we have a treatment, and we want to find out ...
0
votes
0
answers
13
views
Factors in a multi-factor design
I have trouble distinguishing between a factor and a replicate. For instance, in my class notes, there is a question:
To study the variability in CO2 emission rate by global regions 4 countries: US, ...
0
votes
0
answers
22
views
How to handle multicollinearity with varying length and type of conjoint treatments?
We ran a complicated experiment and are struggling to build a linear model that estimates everything we're interested in. We showed each person a description of a product (for illustration, let's say ...
2
votes
1
answer
90
views
Experiments where the treatment is such that the act of applying the treatment to some people within the group and not others undermines the treatment
Let’s say we have an experiment where we want to apply some treatment to some group to test whether the treatment “works”. But the treatment is such that the act of applying the treatment to some ...
1
vote
0
answers
21
views
How do I calculate effect size in a pretest-posttest three group randomized study for two planned contrasts (ANCOVA)?
I am having a hard time calculating the individual effect sizes for ANCOVA simple contrasts ($1$ vs. $2$ and $1$ vs. $3$) in a randomized pretest-posttest design study with three groups. In the ANCOVA,...
2
votes
1
answer
55
views
Can regression to the mean also impact heterogeneity?
I understand Regression to the Mean (RM) as a naturally (and likely randomly) occurring shift in the scores of a sample of extreme subjects (super high or low scorers) from each measurement time to ...
4
votes
1
answer
127
views
how to deal with treatment dropouts in experimental designs
I conducted a between-subjects experiment with one 3-level factor (high group vs low group vs control group). Because of dropouts from the treatments, the final distribution is
control group: 85 ...
5
votes
1
answer
79
views
Performing a Difference-in-Difference Analysis where the Control Group is Already Treated
I'm involved in a project where the outcome is the proportion of cancer patients who have received surgery. The treatment event is a state-level policy change that mandated moving all these cancer ...
0
votes
1
answer
60
views
How to Include a 'No Shift' Scenario in a Temperature Shift DOE
I'm currently working on a Design of Experiments (DOE) for a process that involves temperature shifts at three different times. My challenge is incorporating a 'No Shift' scenario as one of the ...
2
votes
1
answer
67
views
How do data/observations themselves contribute to our ability to make determinations of whether a hypothesis is true or false?
Let's say that I have a hypothesis, and I'm trying to determine whether the hypothesis is true or false. If I remember correctly, in statistics, the standard philosophical view is that, technically, ...
2
votes
1
answer
80
views
Is it possible to use simulated (bootstrap) A/A tests of historic data to estimate the impact of confounding factors on the treatment effect?
I recently heard a proposal for a method to measure the degree to which confounding variables impact historic results of A/B tests.
In order to ascertain the degree to which confounding has impacted ...
1
vote
1
answer
38
views
How to code a lme, lmer, or glmer model for a split-plot design (ANOVA)?
I have a study where the initial height of seedlings were measured at year 0 and again 11 years after treatments were applied and a net change in height was recorded.
I have a split-plot design as ...
2
votes
1
answer
78
views
Power analysis for a quasi-experimental factorial design 3(manipulated) x 1 (measured) ANCOVA
I'm experiencing an hard time looking for the proper analysis to use to compute the power for an experiment i'd like to implement, so here is my question for you.
I'd like to run a quasi-experimental ...
0
votes
0
answers
23
views
What is the appropriate design for a discrete choice experiment?
I need to develop a discrete choice experiment in R, which is an unlabeled experiment.
However, after delving into the theory behind discrete choice experiments, it remains unclear whether there is a ...
1
vote
0
answers
32
views
Discretization in regression, experimentation, and causal inference as deafult [duplicate]
It crossed my mind that when designing an experiment and you're not interested in NHST but full regression model where coefficients for treatment exposure and relevant covariates are desired, perhaps ...
17
votes
3
answers
760
views
Causal diagrams necessary in randomized controlled trials?
I understand how a properly administered RCT rules out confounders because there are no variables influencing the treatment/control group assignment except randomization (meaning no backdoor paths ...
1
vote
0
answers
59
views
How to mathematically prove the "transitive property of nested predictors"?
QUESTION
I am studying the structure of experiment data sets, and I want to propose a rule that I call the "transitive property of nested predictors".
The general idea is that…
if there are ...
0
votes
0
answers
49
views
Validating my experiment design and Mann-Whitney U test usage
There are four different methods I use to generate solutions for different domains.
To determine which solution is the most useful, I want to conduct an experiment with participants.
As I have some ...
0
votes
0
answers
30
views
Caclulating the standard error of difference in Conditional Average Treatment Effects (CATEs)
Some perceive scientific information (for example, scientific evidence of climate change) as accurate, but others don't.
I want to know under which condition this biased evaluation increases or ...
1
vote
0
answers
43
views
Which field of mathematics should be used to formalize the analysis of a dataset's "geometric" structure [closed]
I want to know if mathematical formalism can be applied to understand the "geometric" structure of an experiment's data set, and esp. to the relationships between its categorical predictors.
...
2
votes
0
answers
29
views
Using Difference-in-Differences With No Natural Groupings
I'm running across a number of analyses at my organization where people are doing difference-in-differences on individual-level data with no natural groupings. For example, one looked at the effect ...
8
votes
1
answer
130
views
Is there a good reason for a lab to repeat experiments instead of conducting a single larger blocked experiment
It is not uncommon for biologists to repeat experiments to confirm initial results. Intuitively this makes some sense but to me seems inefficient and potentially problematic.
(For example, one client ...
0
votes
0
answers
35
views
Appropriate design for two experimental conditions and incomplete blocking
I'm planning to perform an experiment in which I will be assessing the role of two experimental variables (A: 6 levels, B: 4 levels) on a defined outcome which, however, might be affected by external ...
0
votes
0
answers
8
views
Conceptual question on Recognition vs conformation
So, this is a conceptual question on the design of a model. I am getting started with ML and on a slight time crunch to get some results. My problem is not a classical recognition problem. It’s more ...
5
votes
0
answers
22
views
Use of other ICU patients as control group
I am reviewing an article that is not yet published, so I cannot be too specific.
Briefly, the authors did an unmatched case-control study. The cases were patients admitted to the ICU for a specific ...
0
votes
0
answers
25
views
Testing correlations between different varibles from two measurements
The field study investigates the relationship between root traits and plant performance traits in 12 genotypes, using a randomized complete block design with four replications per location.
Root ...
0
votes
0
answers
27
views
How to choose the sample size
How do I choose a sample size for a multiple factors irrigation experimental research? Here is what I mean, I want to compare the effect of precision irrigation and conventional irrigation on crop ...
1
vote
0
answers
23
views
Mixed effects model in R with subsampling in replicates
I am working on a mixed effects model in r where I have 2 treatments, 5 replications and subsampling with each site. I dont have access to the raw subsampling data though, I just have the mean effect ...
0
votes
0
answers
12
views
How to adjust blocks in a 2x2 factorial experiment (Minitab)?
I'm currently in the planning phase of an experiment involving 2 factors, each with 2 levels, and 3 replications per combination, resulting in 12 experimental runs. In Minitab, the suggested blocks in ...
6
votes
2
answers
125
views
Designing an experiment to assess the form of a distance measure
BACKGROUND: I do research in budget-proposal aggregation: each person reports their ideal budget-allocation, and a central authority has to decide on an actual budget-allocation. An important ...
4
votes
1
answer
26
views
AB Testing with and without counterfactuals
My boss wants us to run a simple experiment where we manipulate the search results on our website. He is insisting that we must have counterfactuals for the results to be valid. The way I see ...
2
votes
0
answers
44
views
Validating power analysis
I'm currently running some experiments in web. The main idea is to validate the impact in some conversion metrics. The typical funnel we have is: Traffic -> Signup -> some Activation event.
Our ...
4
votes
1
answer
68
views
How does G*Power calculate sensitivity in z-test for two independent proportions
I am trying to recreate sensitivity calculation in Python by applying the same methodology as used in G*Power
z-tests > Proportions: Difference Between Two Independent Proportions > Sensitivity: ...
0
votes
0
answers
9
views
What is the relationship between maximum entropy sampling and bayesian d-optimal design?
In regards to optimal design, specifically sequential design or active learning for parameter inference (e.g. linear regression). As I understand it, Bayesian D-optimal design maximises the expected ...