Questions tagged [experiment-design]

The study of how to structure an information-gathering exercise where variation is present.

Filter by
Sorted by
Tagged with
0 votes
1 answer
32 views

Within subject experiments done by (some) Psychologists

Thanks in advance for any responses or resources you can link me to. I've had more of an Econ/Stats training and recently I've been working with some people who have a Psych background at work. They ...
DirichletC's user avatar
0 votes
1 answer
17 views

Relative importance of equal sample size vs equal sample density

I am conducting a vegetation study of the effects of prescribed fire. I have an experimental area that will be burned and a control area that will not be burned. The experimental area has three types ...
ia200's user avatar
  • 123
1 vote
1 answer
57 views

Which statistical approach is best for diverse conversion rates in an A/B test?

Our software startup builds chat bots for ecommerce websites. The chatbot talks to customers that open the chat bot, and has the goal of closing the sale with the store’s main product. We have about ...
Rage's user avatar
  • 111
3 votes
1 answer
45 views

Understanding "Multiplication/ Group Operation" in Fractional Experiments

There's a group operation, at least of sorts, in Fractional Factorial design that I'm trying to understand. For definiteness, let's say we have 3 factors; A,B,C , at two levels each . Please critique ...
MSIS's user avatar
  • 549
4 votes
2 answers
209 views

Double Blind Studies & Confounders

I'm currently reading The Book of Why, and on pg 147-149 it talks about double blind RCT being very highly regarded in experiments, especially in the way it address confounders. In theory, I can ...
ron burgundy's user avatar
0 votes
0 answers
35 views

How can I use GLMM for my data?

So let me tell you a little about my data I have two columns of transects and plots as random effects I have an Elevation column as a fixed effect, My response variable is low and high grazing data ###...
Zeinab's user avatar
  • 1
2 votes
0 answers
37 views

What is the normalized frequency distribution for the shortest tour length of identical circles in a larger circle? [migrated]

Suppose that I have a a circle of radius, $R$ with $n$-many non-overlapping circles in it where each circle has a radius, $r$. The first block of code below generates random circles of the same radius ...
Teg Louis's user avatar
  • 121
2 votes
1 answer
63 views

Experimental design: What gives you more statistical power: Increasing number of sub samples or replicates?

Say I have an experiment with a binary response: 0 or 1. Within each experiment, I put X number of individuals in a group (X sub samples) and measure the proportion that has a response of 1 (e.g. 2/4)....
scott.pilgrim.vs.r's user avatar
1 vote
0 answers
45 views

How to proceed when the observed effect is smaller than the Minimum Detectable Effect?

Given an A/B test with an already defined sample size and variance, a power analysis was conducted to estimate the Minimum Detectable Effect (MDE) of a proportion. After the experiment was concluded, ...
loglossbb's user avatar
0 votes
0 answers
15 views

Good way to select reference and/or performing normalization of data points across experiments?

I have a set of objects S1 = {a1, b1, c1, d1, ..., z1} that perturbed a reference object q and were tested for effects in ...
neo4k's user avatar
  • 1
2 votes
0 answers
20 views

What test of reliability for repeated measurements of the same scale?

I need to ask you for help on what method to use to calculate the reliability and validity of a scale that I measured repeatedly on the same subjects during an experiment. Let me explain: normally ...
Ric87's user avatar
  • 71
0 votes
0 answers
30 views

Formulating a linear mixed model with multiple random effects when one is not independent

I have a randomized complete block design. I have formulated linear mixed models using R, the package lme4, and the function lmer for all measurements except for one, where I am stuck on the random ...
Ag123's user avatar
  • 1
0 votes
0 answers
8 views

Exogenous but non-random event for DiD

I have a quick question regarding whether the following event might pose any issues for my study. I am interested in exploring perceptions of climate change. In the early 1970s, former dictator Park ...
chunguc1004's user avatar
3 votes
0 answers
24 views

Regression models as goal metrics for experiments

What are the known issues or pitfalls associated with scoring regression/ML models on observations involved in a randomized experiment (a/b test, etc.)? In a randomized experiment, for most metrics (...
user1993951's user avatar
0 votes
0 answers
16 views

Machine Learning Experiment: should training parameters be fixed for a valid comparisons across model?

I am training an autoencoder three times, each time on a different dataset. The three datasets all have the same number of features, but have vastly different sizes. Assuming one of the datasets is A ...
BovineScatologist's user avatar
0 votes
1 answer
48 views

Usefulness of Latin Square Design

I want to understand the usefulness of the Latin Square Design. Suppose there are two block factors, each with three blocks. There are three experimental groups: A, B, C. One realization of the Latin ...
quibble's user avatar
  • 1,436
0 votes
0 answers
11 views

Is there a best approach for statistical power for field experiment?

I'm setting up a field experiment and want to make a solid plan for the analysis, since that's whereI get stressed in the process. I will deploy 26 pairs of traps. 13 pairs will be in urban forest, ...
E10's user avatar
  • 81
1 vote
0 answers
14 views

Is a permutation test valid in comparing a fraud block policy false positive rate across two different months?

I'm looking to conduct a counterfactual experiment in an e-commerce fraud context. When a fraud policy blocks a transaction you necessarily will never know if it was truly fraudulent. I'm trying to ...
phx's user avatar
  • 11
0 votes
0 answers
19 views

How can I implement a linear mixed model on data where the reference doesn't have repeated measures?

I want to analyze in paired data that has many dependent variables (continuous) and several categorical predictors. The predictors are: Group (patient/control), Treatment (none, t1, t2, t3), and Gene (...
maglorismyspiritanimal's user avatar
3 votes
1 answer
190 views

Reduce the model sequentially

I was given an ANOVA table and asked to reduce the model sequentially. I searched the online resources say: When reducing the model sequentially, you typically start by assessing the significance of ...
Matata's user avatar
  • 671
2 votes
1 answer
48 views

Statistical Test or Regression for Intervention Effectiveness?

I'm looking to model the effectiveness of an intervention on human behavior. I have control groups and treatments groups for the outcome of interest and pre and post measurements (just 1 for each) for ...
Statsnoob's user avatar
1 vote
0 answers
21 views

Why can't interaction be measured with a complete block design? [duplicate]

I am struggling to understand why interaction cannot be measured in a CBD. While many sources say it is because there is only one observation per block/treatment combination, could you not compare the ...
Laura Edwards's user avatar
0 votes
0 answers
33 views

Looking for an intuitive explanation of D-Criterion for Optimal Design Problem

I know only a little about Fisher information and optimal experimental design, but I'm trying to better understand the subject. If I have an experiment composed of a single detector and my detector ...
David G.'s user avatar
  • 117
0 votes
1 answer
22 views

Show conditional independence in randomized experiment

I'm new to causal inference, and I have a question regarding a potential causal relationship. I'm wondering if the following assumption holds: $(Y_i(1), Y_i(0), X_i) ⊥ Z_i -> (Y_i(1), Y_i(0)) ⊥ Z_i ...
user408381's user avatar
2 votes
1 answer
46 views

Regression using unordered combinations, sign of predictor depends on order

First time poster here. Is there anything wrong with the following regression approach? EDIT: After following the advice of the accepted answer to include the multiple-membership effect, I wanted to ...
Evan Stegner's user avatar
0 votes
0 answers
25 views

What is the correct way to analyze this data in a mixed model approach?

I have an experiment with rates of an waste in complete randomized blocks designs, with evaluation of soil fertility and tree heights/diameters over the years. Which is the best model to fit this data?...
ntrentin's user avatar
7 votes
2 answers
586 views

average revenue per user increases each country but decreases when combined

I'm trying to figure out what could be some reasons when we do the AB testing we're seeing average revenue per user increases for each country but when we calculate the overall average revenue per ...
ggggogloria's user avatar
5 votes
1 answer
93 views

How is it reasonable that randomised controlled trials can be used to perform causal inference?

I understand that randomised controlled trials (RCTs) are used to perform causal inference, but I'm a confused about how this is reasonable. Let's say that we have a treatment, and we want to find out ...
The Pointer's user avatar
  • 1,892
0 votes
0 answers
13 views

Factors in a multi-factor design

I have trouble distinguishing between a factor and a replicate. For instance, in my class notes, there is a question: To study the variability in CO2 emission rate by global regions 4 countries: US, ...
Matata's user avatar
  • 671
0 votes
0 answers
22 views

How to handle multicollinearity with varying length and type of conjoint treatments?

We ran a complicated experiment and are struggling to build a linear model that estimates everything we're interested in. We showed each person a description of a product (for illustration, let's say ...
Tia's user avatar
  • 51
4 votes
1 answer
95 views

Experiments where the treatment is such that the act of applying the treatment to some people within the group and not others undermines the treatment

Let’s say we have an experiment where we want to apply some treatment to some group to test whether the treatment “works”. But the treatment is such that the act of applying the treatment to some ...
The Pointer's user avatar
  • 1,892
1 vote
0 answers
24 views

How do I calculate effect size in a pretest-posttest three group randomized study for two planned contrasts (ANCOVA)?

I am having a hard time calculating the individual effect sizes for ANCOVA simple contrasts ($1$ vs. $2$ and $1$ vs. $3$) in a randomized pretest-posttest design study with three groups. In the ANCOVA,...
Eniko's user avatar
  • 11
2 votes
1 answer
55 views

Can regression to the mean also impact heterogeneity?

I understand Regression to the Mean (RM) as a naturally (and likely randomly) occurring shift in the scores of a sample of extreme subjects (super high or low scorers) from each measurement time to ...
Simon Harmel's user avatar
4 votes
1 answer
131 views

how to deal with treatment dropouts in experimental designs

I conducted a between-subjects experiment with one 3-level factor (high group vs low group vs control group). Because of dropouts from the treatments, the final distribution is control group: 85 ...
dondu's user avatar
  • 41
5 votes
1 answer
81 views

Performing a Difference-in-Difference Analysis where the Control Group is Already Treated

I'm involved in a project where the outcome is the proportion of cancer patients who have received surgery. The treatment event is a state-level policy change that mandated moving all these cancer ...
user8408275's user avatar
0 votes
1 answer
64 views

How to Include a 'No Shift' Scenario in a Temperature Shift DOE

I'm currently working on a Design of Experiments (DOE) for a process that involves temperature shifts at three different times. My challenge is incorporating a 'No Shift' scenario as one of the ...
Megazord's user avatar
3 votes
1 answer
69 views

How do data/observations themselves contribute to our ability to make determinations of whether a hypothesis is true or false?

Let's say that I have a hypothesis, and I'm trying to determine whether the hypothesis is true or false. If I remember correctly, in statistics, the standard philosophical view is that, technically, ...
The Pointer's user avatar
  • 1,892
2 votes
1 answer
84 views

Is it possible to use simulated (bootstrap) A/A tests of historic data to estimate the impact of confounding factors on the treatment effect?

I recently heard a proposal for a method to measure the degree to which confounding variables impact historic results of A/B tests. In order to ascertain the degree to which confounding has impacted ...
McGez's user avatar
  • 91
1 vote
1 answer
41 views

How to code a lme, lmer, or glmer model for a split-plot design (ANOVA)?

I have a study where the initial height of seedlings were measured at year 0 and again 11 years after treatments were applied and a net change in height was recorded. I have a split-plot design as ...
kryogenic7's user avatar
2 votes
1 answer
82 views

Power analysis for a quasi-experimental factorial design 3(manipulated) x 1 (measured) ANCOVA

I'm experiencing an hard time looking for the proper analysis to use to compute the power for an experiment i'd like to implement, so here is my question for you. I'd like to run a quasi-experimental ...
GT87's user avatar
  • 143
0 votes
0 answers
25 views

What is the appropriate design for a discrete choice experiment?

I need to develop a discrete choice experiment in R, which is an unlabeled experiment. However, after delving into the theory behind discrete choice experiments, it remains unclear whether there is a ...
Amy95's user avatar
  • 1
1 vote
0 answers
32 views

Discretization in regression, experimentation, and causal inference as deafult [duplicate]

It crossed my mind that when designing an experiment and you're not interested in NHST but full regression model where coefficients for treatment exposure and relevant covariates are desired, perhaps ...
jbuddy_13's user avatar
  • 3,020
18 votes
3 answers
786 views

Causal diagrams necessary in randomized controlled trials?

I understand how a properly administered RCT rules out confounders because there are no variables influencing the treatment/control group assignment except randomization (meaning no backdoor paths ...
RobertF's user avatar
  • 6,104
1 vote
0 answers
59 views

How to mathematically prove the "transitive property of nested predictors"?

QUESTION I am studying the structure of experiment data sets, and I want to propose a rule that I call the "transitive property of nested predictors". The general idea is that… if there are ...
Chris Science's user avatar
0 votes
0 answers
49 views

Validating my experiment design and Mann-Whitney U test usage

There are four different methods I use to generate solutions for different domains. To determine which solution is the most useful, I want to conduct an experiment with participants. As I have some ...
Max Bragilovski's user avatar
0 votes
0 answers
30 views

Caclulating the standard error of difference in Conditional Average Treatment Effects (CATEs)

Some perceive scientific information (for example, scientific evidence of climate change) as accurate, but others don't. I want to know under which condition this biased evaluation increases or ...
Jin's user avatar
  • 1
1 vote
0 answers
43 views

Which field of mathematics should be used to formalize the analysis of a dataset's "geometric" structure [closed]

I want to know if mathematical formalism can be applied to understand the "geometric" structure of an experiment's data set, and esp. to the relationships between its categorical predictors. ...
Chris Science's user avatar
2 votes
0 answers
29 views

Using Difference-in-Differences With No Natural Groupings

I'm running across a number of analyses at my organization where people are doing difference-in-differences on individual-level data with no natural groupings. For example, one looked at the effect ...
kypc's user avatar
  • 21
8 votes
1 answer
132 views

Is there a good reason for a lab to repeat experiments instead of conducting a single larger blocked experiment

It is not uncommon for biologists to repeat experiments to confirm initial results. Intuitively this makes some sense but to me seems inefficient and potentially problematic. (For example, one client ...
George Savva's user avatar
  • 2,375
0 votes
0 answers
37 views

Appropriate design for two experimental conditions and incomplete blocking

I'm planning to perform an experiment in which I will be assessing the role of two experimental variables (A: 6 levels, B: 4 levels) on a defined outcome which, however, might be affected by external ...
Mabri's user avatar
  • 21

1
2 3 4 5
40