Questions tagged [experiment-design]

The study of how to structure an information-gathering exercise where variation is present.

Filter by
Sorted by
Tagged with
4
votes
0answers
56 views

D-Optimal Criteria Vs Differential Shannon Entropy

I want to understand how minimizing the determinant of the information matrix is equivalent to maximizing the differential Shannon entropy? A similar question was posted in Math SE but hasnt been ...
2
votes
1answer
152 views

Bayesian A/B test analysis for non-binary actions

When analyzing experiments with binary action (for example, will a user convert or not), then Beta-Binomial distribution is apt for that. For example, to model Click Through Rate (CTR): ...
2
votes
1answer
38 views

Do I have to randomize all drug trials?

This thought experiment I came up with has been bugging me. Let's say I want to test the effectiveness of Drug A vs Drug B on reading speed (no placebo necessary). I KNOW for a fact that the effects ...
0
votes
1answer
27 views

Conditional treatment effects for RCTs

I have randomized controlled trial data with one control and two treatments. Since the DV is continuous, I estimate the treatment effect using an lm model, while ...
0
votes
0answers
22 views

Why confounding factor has to correlated with independent variable?

One assumption of linear regression is the independent variables should be independent of each other. But in experimental design, when we build a linear regression to measure the causal impact of the ...
5
votes
2answers
11k views

What's the difference between a randomized block design and two factor design?

What's the difference between a randomized block design and a two factor design? They both use two-way ANOVA, and your blocks can be your factor.
1
vote
1answer
138 views

What is the meaning of “each treatment ” in BIBD?

(I am restating the question again in a more clear manner, as it was not clearly written last time) My doubt is regarding Balanced Incomplete Block Design (BIBD) Consider the following design with ...
1
vote
1answer
40 views

How is it decided to use Difference in Differences, Regression Discontinuity or both together (if possible) according to the policy change?

I am quite new at (quasi) natural experiment analysis, so please bear with me, and got some questions. In my research, I am planning to use one policy change which happened 2002 and treated firms are ...
0
votes
1answer
132 views

Partially crossed design and how to apply the mixed model?

I have a list of patient samples that were tested at 3 clinical sites; for site 1, reagent lots 1 and 2 were applied; for site 2, reagent lots 1 and 3 were applied; for site 3, reagent lots 2 and 3 ...
1
vote
2answers
30 views

Can a “crossed” experimental design compare a continuous and categorical variable?

Would it be incorrect to refer to an experimental design as "crossed" if one of the treatments is a continuous variable rather than a categorical variable? The continuous variable covers roughly the ...
3
votes
2answers
60 views

Optimal unbalanced design for A/B test

I want to find the right way to distribute variants (control/test) in A/B-test with the constraint of no more 10 percent of population in test group. The test is t-test for continuous metric. I can ...
0
votes
2answers
27 views

Apply normality assumption on limited sample size

I now have a measurement which only gives me: i) the range of values (continuous in nature), say, 110.0 to 160.0 without the access to the actual underlying data from which these statistics are ...
2
votes
1answer
42 views

What to do if the range of an input factor changes after running the experiment?

I am gaining my first experiences in Designing Experiments and here where I am stack: An experiment has been run with some continuous input factors: x1, x2, ..., x5. The goal of the experiment was to ...
0
votes
0answers
17 views

What kind of test for percentages of different groups?

I've set up a little experiment using 5 different treatments (including the control group) for control of crown rot in harvested bananas. For each treatment, I took around 18-20 banana clusters and ...
51
votes
11answers
5k views

Famous easy to understand examples of a confounding variable invalidating a study

Are there any well-known statistical studies that were originally published and thought to be valid, but later had to be thrown out due to a confounding variable that wasn't taken into account? I'm ...
1
vote
1answer
26 views

Sample Size Calculation considering Gradual Improvements to Treatment Group

I have an experiment with two equally sized groups. These groups contain machines and the metric being measured is the failure rate of the of the groups. In a typical experiment setting, the test ...
1
vote
1answer
27 views

Which additional experiments would best improve the model?

I have been given some data on a biotechnological process stemming from a multitude of different (kind of independent) experiments. The process has a lot of potentially relevant inputs (temperature, ...
0
votes
0answers
26 views

experimental design and linear regression

In an experimental design, randomly assigned participants into treatment group and control group and there is no statistical difference(using t-test) between two groups. The treatment group was ...
0
votes
0answers
16 views

Sample size calculation for AB testing in offline scenario

I have sales data for 500 stores and lets say I want to run an AB test on a sample of test and control stores with the objective of getting statistically significant increase in sales(lift). My ...
0
votes
0answers
37 views

List Experiment “ictreg.joint” error: “log-likelihood is not monotonically increasing”

I'm trying to use "ictreg.joint" function in "list" package in R by Imai and Blair to use the predicted responses from the list experiment as the explanatory variable to analyze an outcome variable. ...
0
votes
1answer
154 views

Using a model to select between a/b in a a/b test

I am wondering what the best way is to model an A/B test. Meaning, if I run an a/b test, I want to run a model to determine if A or B is better for each individual. A/B would be assigned randomly (50%...
0
votes
1answer
21 views

MDE minimal detectable effect

I m not able to figure out how to calculate MDE(minimal detectable effect) when designing an experiment. I see that there are calculators to do it for you like this one: https://bit.ly/2XgMjld For: ...
0
votes
0answers
5 views

Interaction Effects, Zero or non-Zero in 2 factor factorial design?

In a 2 factor factorial design it is common to use the following linear model and the associated notation: $y_{ijk}=\mu+\alpha_{i}+\beta_{j}+(\alpha\beta)_{ij}+\epsilon_{ijk}$ $(\alpha\beta)_{ij}$ ...
0
votes
0answers
22 views

detecting correlation between and significance of non-linear model parameters

I have a set of observed data y,obs that I am modelling using a non-linear model (currently in EXCEL though I also have a Python model). There are 4 input ...
4
votes
2answers
67 views

How to include an interaction with sex

I am performing linear regression on a dataset and want to include an interaction between sex and another covariate. However, in my model sex is coded as $0$ for female and $1$ for male. If I have ...
2
votes
0answers
39 views

How should I analyse the results of my experiments with respect to their hyper-parameters?

I trained (and tested) a CNN with the same architecture on the same task (MNIST). However, I trained it several times: one for each combination of hyper-parameters. Each model is trained and tested ...
2
votes
1answer
181 views

How to choose/select an experimental design

Is there a rule for selecting which experimental design one should carry out? I am new to statistics/ED and sometimes I find myself asking which design is best to use. My general guidelines have been ...
0
votes
2answers
145 views

Different groups, same intervention. What design is this?

I'm trying to figure out the best way to analyse the following clinical trial: Group1: Healthy individuals Group2: Diseased individuals Both groups follow the same intervention and are tested pre- ...
0
votes
0answers
10 views

How can I apply a design effect of 2 on my fisher's test and/or regression?

I'm trying to use R or statistical analysis on a multicluster sampling and specifically to inform it that I have a design effect of 2. I've familiarised myself with the survey package and tried to ...
0
votes
0answers
19 views

How to deciding on the number of experimental runs in R for orthogonal fractional factorial design?

Presently I am working on designing a questionnaire for my discrete choice experiment. I want to generate an orthogonal fractional factorial design for the following problem- The respondent has to ...
1
vote
0answers
35 views

Experiment design advice

I’d like to understand a good experiment design for the following example: Say sales people are trying to find the ideal time (in days) to send a follow up phone call to increase conversion (a sale). ...
4
votes
1answer
106 views

How to analyze factors that have missing by design values for certain observations?

I am using an analysis for an experiment that I consider suboptimal. I am asking for any suggestions to improve this. I will briefly outline the experiment and what I’ve done, then ask your advice. ...
2
votes
1answer
40 views

Generating a full rank model matrix in R

For a particular tool I need a model matrix which allows me to build desired contrasts and which is a full rank, i.e. no columns are linear combinations of other columns. The experimental design ...
1
vote
0answers
10 views

Factor analysis with missing data

I am looking for methods that allow me to do an exploratory factor analysis with missing data. The reason I need to account for missing data is that we will try to collect a large number of variables ...
1
vote
1answer
25 views

How to interpret coefficients of a cross-lagged panel analysis

I'm trying to understand the coefficients of this cross-lagged panel analysis. Are they correlation coefficients? are they regression coefficients? I'm not sure how can I interpret them if I have no ...
1
vote
0answers
22 views

Compare performance of libraries - test selection

I am trying to perform a comparison between three implementations of the same algorithm but using different libraries. However, I am not sure about my methodology. I am measuring elapsed time as a one ...
0
votes
0answers
17 views

$\mathbb{G}=(1,B,T,B \times T)$ is an orthogonal design

I have an orthogonal system with 2 factors, B and T. I know that B and T are orthogonal geometric. But I also need to show that that the system $\mathbb{G}=(1,B,T,B \times T)$ is an orthogonal ...
1
vote
0answers
8 views

Aggregate effects of randomized experiment

Consider the following two-equation model for observational data from a randomized experiment: (1) $y_{ij} = \alpha + \beta D_{ij} + \gamma X_{j} + \varepsilon_{ij}$ (2) $D_{ij} = \gamma + \delta ...
50
votes
8answers
7k views

Statistical tests when sample size is 1

I'm a high school math teacher who is a bit stumped. A Biology student came to me with his experiment wanting to know what kind of statistical analysis he can do with his data (yes, he should have ...
1
vote
0answers
43 views

What is the dependent variable in an Item Count Technique Regression (List experiment)

I try to understand the logic behind an Item Count Technique Regression (Imai 2011; Blair & Imai 2012). I am especially interested in the ML estimator for the multivariate regression analysis (...
0
votes
0answers
21 views

Randomized Block Design - Feature is Now Insignificant

I'm working with COVID-19 data for a class project. Our initial randomized design revealed that the factor Income played an insignificant role in predicting the response variable Mortality Rate (p > 0....
1
vote
0answers
13 views

How are terms in SS (corrected sum of squares) independent?

What does the highlighted test mean? How are the terms not independent? Can someone elaborate in detail?
1
vote
0answers
26 views

help requested for mirror polishing data set

In space satellites, mirrors are used to guide incoming light to detectors measuring the features of interest. It is crucial that these mirrors are very smooth so that they do not unduly scatter the ...
0
votes
0answers
38 views

Optimal point pairs for estimating the distance function in a metric space

I’m looking for the name of the methods or subfield that deals with problems like this: Assume that we have $n \times m$ points over a regular grid.They are embedded into a metric space. We can get ...
2
votes
1answer
75 views

For a permutation test, why does an upper bound for the number of permutations $B$ not depend on the sample size?

I am trying to find what permutation size $B$ I should be using for conducting a permutation test. Currently I have two samples which are normal, same mean but different variance. The sample sizes are ...
3
votes
1answer
67 views

Book/resource recommendation for design and analysis of clinical trials

I will be transitioning from a role analyzing purely observational epidemiological data to a role analyzing and designing clinical trials. I was wondering if you may have any recommendations for books ...
3
votes
1answer
37 views

Am I violating any causal inference assumptions?

Consider a large-scale AB test with two variants: control and treatment. Assume we want to compare the two variants with respect to the following metric: "Proportion of users who streams more than X ...
1
vote
0answers
50 views

Is there value in assigning a dimension-specific weighting when calculating the main effect?

I've been looking into factorial designs and noticed an interesting property. Is this simply a mathematical trick or can we attribute some 'meaning' to it? Say we have a $2^k$ design where k = 3 here....
1
vote
0answers
28 views

Matched Pairs design definition

A horticulturist is planning a greenhouse experiment with 20 individually potted tomato plants to compare two fertilizer treatments. Ten plants are randomly assigned to fertilizer A, and the remaining ...
2
votes
0answers
10 views

In A/B test, how to deal inconsistency between unit of experiment and unit of analysis in the experiment design?

In the experiment, the metrics we care about is clicks per page (#clicks / #pages). When performing the experiment, the unit of diversion we randomized on is "user" instead of "page". I'm confused ...

1
2
3 4 5
29