Questions tagged [experiment-design]

The study of how to structure an information-gathering exercise where variation is present.

Filter by
Sorted by
Tagged with
5
votes
3answers
9k views

Calculate number of needed simulations

First I have to apologize for any uncorrect naming or categorisation of my question, as I am an electrical engineer rather than a mathematican. I have a simulation that outputs for a given number of ...
3
votes
3answers
901 views

Participant identifier for a field study in a sensitive work environment

We are doing an evaluation in different organizations and need to get data from workers about their work behavior. As this information is something that could get workers in trouble, privacy is an ...
1
vote
0answers
1k views

How many degrees of freedom with many subjects and many tests?

Lets say I run an experiment testing accuracy of hitting targets of two different sizes. I have 10 participants and 40 targets, 15 small and 25 large. I then have the number of misses for each target ...
3
votes
2answers
473 views

Experimental design: what to do when factors cannot be held at constant levels?

I am looking at some data with a 5 factors and a response variable. The experiment was designed as a factorial experiment, with observations at different levels of each factor. One thing that I have ...
2
votes
0answers
1k views

2x2x2 plus control group: nested ANOVA?

I recently ran an experiment assessing the impact of three system factors (factor A; factor B; factor C) on user attitudes (4-item scale). Each system factor had two levels (high vs. low). As a result,...
0
votes
1answer
209 views

What statistical test should be used to accomplish the objectives of the experiment

I'd like to have your feedback to analyze my data from the following experiment. Experiment Information A total of 30 (15 from one city and other 15 from another city) slaughtered animals were ...
1
vote
1answer
3k views

Categorical dependent variable in between subject design

My colleague has an experiment with a 2x2 all between subjects design with categorical variables predicting a three-level categorical dependent variable (choise between A, B, C/ coded as -1 0 1). Is ...
2
votes
0answers
157 views

Weighting a control group

In reviewing some market research, I came across a study which decided to weight the results/outcomes of the control group so the distribution of participants in the control sub-groups (0,1,2,3,4,5 - ...
1
vote
0answers
632 views

Loss functions, decision theory for hyperparameters, or estimating the variance of an unknown prior

How I got here I was originally interested in point estimation and the Bayes risk of some distribution $\pi$ $$ r(\pi) = \mathbb{E}_{\pi(x)}[ \mathbb{E}_{\Pr(y|e,x)}[L(x,\hat x(y|e))]], $$ where $...
3
votes
1answer
8k views

How to perform a t-test to compare treatment and control groups when the experiment has been replicated three times?

I have data from a experiment that has been repeated 3 times under the same condition. Each time, 5 measurements of each of 2 groups (a control and a treatment) were taken. I want to run a t-test on ...
2
votes
1answer
595 views

How to deal with a 2x2 experimental design if the dependent variable is dichotomous?

I am assessing the acceptability of a sentence in a within-subjects design with two crossed factors (type of sentence, missing word). After reading each sentence, the participants answer a simple yes/...
2
votes
0answers
91 views
2
votes
1answer
121 views

Change in gene expression over time for different environments when each individual can only be sampled once?

A colleague has collected data on plant root gene expression. Gene expression is expected to change over time, and be dependent upon the environment that each plant is grown in (i.e fertilizer mix). ...
3
votes
1answer
2k views

Statistical tests for low sample numbers in both indendent groups (n=3 and n=1)

I've been asked to analyze some data for an experiment that's already been done. 15 rats were raised in 5 groups of 3 No treatment Saline treatment (control drug is delivered via saline) Drug ...
2
votes
2answers
2k views

Good starting book on psychometrics

I am looking for a nice concise statistics book that summarizes how to do the experimental designs properly, and maybe some algorithms people in the field have discovered. What are the best books for ...
1
vote
1answer
515 views

Getting started with multivariate testing

I'm looking for the best combination of variables to result in a click to the next page of my website... For example the size and color of a submit button. I have a system that currently tests each ...
5
votes
2answers
2k views

Analyze and visualize participants response towards particular condition

Method: I presented 15 participants with audiovisual clips. There were six different clips I presented, and for each clip there was emotional incongruence between visual and auditory information. The ...
7
votes
1answer
11k views

Stratified sampling with multiple variables?

I don't know much about stats so I'm looking for a starting point here. Any resources or insights would be helpful. I'm conducting an e-learning experiment, in which students watch videos and then ...
3
votes
2answers
279 views

Normalization in pairwise hypothesis testing

I'm reviewing a conference paper (not a stats conference). The authors have done some funny things with their methodology. The experiment setup has 4 treatments, each applied to the same 30 subjects. ...
9
votes
1answer
445 views

Should I re-shuffle my data?

We have a set biological samples that was pretty expensive to obtain. We put these samples through a series of tests to generate data that are used for building a predictive model. For this purpose we ...
6
votes
1answer
146 views

What do you do with your testing data?

Suppose you perform a parameter selection study in an environment with not-so-many data points. You divide the available data into training and testing set and you plan to validate the future model on ...
9
votes
1answer
492 views

Why is running split tests until statistically significant a “bad thing”? (Or is it?)

I read this article on "how not to run an A/B test". And I still don't understand what exactly the author's reasoning is. Can someone dumb it down for me? I think what it might be saying is that ...
1
vote
3answers
1k views

How to identify suitable variables to assess confounding, mediation and effect modification?

Imagine that you are planning a study about risk behaviours among HIV positive injecting drug users. All the individuals included in the sample are injecting drugs and all are HIV positive. The main “...
5
votes
1answer
762 views

Data analysis for mixed within-subjects/between-subjects design?

I have run a survey with many respondents. There was a pool of questions, and each respondent was asked 3 of the questions (chosen uniformly at random and independently for each respondent). Now I'm ...
2
votes
0answers
205 views

Analysis of multimodal metric response variables linked to a categorical variable

my level of stats is kind of restricted to some undergraduate courses. Anyhow I received some data from a planned experiment. Basically it is about to test the shear strength (interval scale) for ...
3
votes
0answers
141 views

Estimating parameters in a model with a periodic design parameter

We recently studied a model with a likelihood function of the form $$\Pr(d_i|\omega,t_i)=(1-d_i)+(2d_i-1)\cos^2(\omega t_i),$$ where $d_i\in\{0,1\}$, $\omega$ is an estimation parameter, and $t_i$ ...
6
votes
2answers
579 views

Review of case control matching algorithms?

Can you suggest a good review of case control matching algorithms? Algorithms that can be used to set up the matched pairs of one case and one control, or matched blocks of a case and multiple ...
0
votes
0answers
120 views

How to choose best value of 2 parameters simultaneously?

I have devised an algorithm that takes as input a dataset $s$ and two parameters, $k$ and $d$. $k$ is discrete while $d$ is categorical. My algorithm outputs a vector $v$ of length 20 which contains ...
8
votes
2answers
2k views

How to calculate sample size for simulation in order to assert some level of goodness in my results?

I am a stats newbie, so apologies in advance if I'm asking a braindead question. I have searched for answers to my question, but I find that many of the topics are either too specific, or quickly go ...
5
votes
2answers
90 views

Is it a problem that the sizes of randomized groups can differ?

My website has advertising for various companies - say 10,000 companies. Each company has it's own page but all the pages are currently similar except by which company is on the page. A success/...
5
votes
3answers
165 views

Flipping test and control groups

Let's say you're running an A/B test on a website. Let's also say the standard methodology of randomly displaying one version or the other is unusable due to SEO hand-waving. The particular page ...
5
votes
1answer
196 views

Testing significance of factors and covariates along with modeling causality among responses

I'm wondering how to test the significance of factor(s) and/or covariate(s) along with modeling the causal relationship among responses. Let me explain this with a concrete example. Example: Suppose ...
2
votes
1answer
98 views

Summary statistic for an error measurement

I have $n$ error measures that can be positive or negative values. Negative values are more important for my experiment than positive values. Of course, taking the mean of the $n$ values doesn't make ...
3
votes
1answer
770 views

How to analyse pre-post quasi-experimental study with an additional baseline measure?

I am having a hard time with what seems to be a very basic research design issue. The study has 3 variables: Self-control and ...
1
vote
0answers
469 views

How to deal with this design?

I have an experiment with 3 biological factors (dev, dev_stage; reg, brain_region; gen, ...
3
votes
1answer
775 views

How to test whether percentage is significantly greater than 50% after averaging across three experiments?

Context: I ran 3 experiments comparing in each 2 techniques. In each experiment participants had simply to express a preference about which of the 2 techniques they thought was clearest to understand....
4
votes
1answer
351 views

AB Multi-variate testing - when can I eliminate worst performing variants?

I'm running AB tests on my website homepage, with 8 different variations. You can read about the purpose of the test here if you need to (not essential) - http://westiseast.co.uk/blog/ab-split-testing-...
2
votes
1answer
95 views

How to design a study and test the effect of dosage level on cure probability?

My goal is to quantify dosage impact on cure probability for different patients. Let's suppose I have N patients with their charateristics such as age, gender, weight.... Also let's assume there is a ...
2
votes
3answers
390 views

Testing a binomial model

I have a computational experiment that takes a size parameter $s$ and returns true or false. It has been suggested that the result of the experiment is independent of $s$, but I don't think so. I ...
2
votes
3answers
373 views

Mouse models - 'replicates' and analysis

I'm hoping to hear from someone who has worked on mouse models or similar biological analyses where there is a tendency to run 'replicates' of an experiment. I know multiple testing is a sizeable ...
1
vote
0answers
94 views

Formulate hypotheses when $\mu_A < \mu_B$ is different from $\mu_A > \mu_B$

I'm testing two different settings (say 'A' and 'B') for a machine. From physical arguments, I know that $\mu_B >= \mu_A$. If really $\mu_B < \mu_A$, there's something fundamentally wrong with ...
6
votes
1answer
200 views

How to predict how much data to collect?

I'm currently in the planning stages of an experiment that I hope will detect a 0.155% signal variation (relative magnitude) within a reasonable time frame (less than 6 months ideally). I've ...
2
votes
2answers
184 views

Is this a repeated measures experiment?

We conducted an experiment on hens with seven treatments (medicines) replicated three times. Every treatment was implemented on a flock of fifteen young hens and three of these treated hens were ...
2
votes
1answer
574 views

What is the proper ratio of mean squares for a two-level nested ANOVA?

I would like to analyse the following type of experiment using a two-level fully nested ANOVA. We have mice of two genotypes (Group factor) and for each of the genotypes we take a certain number of ...
6
votes
2answers
499 views

Which statistical test should I use for my experiment on aggressive interactions in killifish?

I am doing a project on sexual selection (male-male competition) in the turquoise killifish Nothobranchius furzeri. There are two morphs of male in the population from which my fish are obtained ...
1
vote
1answer
1k views

How to describe a design with a mix of experimental conditions, predictor variables, and multiple outcome variables?

My research design is as follows: I have these Between Subjects IVs: Experiment Condition - 5 levels Facebook User status - 2 levels (yes/no) My supervisor also wants me to see if there are ...
3
votes
1answer
1k views

Individuals standard deviations and/or standard errors for groups after implementing ANOVA?

I've seen couple of articles reporting individuals standard deviations and/or standard errors for groups even after implementing ANOVA. My understanding is that groups SE's should be based on ...
1
vote
2answers
183 views

Optimal blind poll construction

This is a question strongly related to Cauchy "characters". I'm constructing a 4 question canvassing questionnaire that will tell the likely voter being contacted which of the presidential candidates ...
5
votes
1answer
358 views

Number of times to run a lengthy experiment

I have this long running experiment. Each time I run it I get a new goodness value, since the algorithm has random variables in it. So I need to report the mean and the std of some n runs. What should ...
3
votes
1answer
331 views

How to predict and optimize a queue?

I am trying to determine if I can model my system as a M/M/1 queue and if so do the numbers I get from it help me at all. I can model my system like this: System Description A. I want to spec out ...