Questions tagged [replicability]
The extent to which subsequent studies of the same phenomenon reproduce the results obtained in the original study.
31
questions
2
votes
1
answer
60
views
How to Replicate result that probabilities in included table are different 7 times?
Question: In the table at the bottom of this question, the total number of times that $\hat{P}(x;z) > \hat{P}_y(x;z) $ and $\hat{P}(y;z)>\hat{P}_x(y;z)$ is 7.
Can someone please replicate this ...
- 219
1
vote
1
answer
50
views
Experiment design useful when objects change / will be replaced in each trial run?
I want to perform some studies on how material thickness, type of material, humidity and temperature influence tensile strength.
While I was thinking about useful designs I was wondering whether a ...
- 3,041
0
votes
1
answer
35
views
What are tests to determine the number of technical replicates neccessary to reach the true mean?
I have the following scenario: I am working with a quite variable material, so I am using technical replicates to estimate the mean sample properties. To test how many replicas I need to converge to ...
- 153
3
votes
2
answers
107
views
Could we estimate replicability of empirical research with conformal predictions?
A review article Threats of a Replication Crisis in Empirical Computer Science reviews reproducibility issues. The authors present distinctions among repeatability, replicability, and reproducibility. ...
- 2,442
2
votes
1
answer
51
views
Can confidence intervals or uncertainty intervals provide information on replication? [duplicate]
Let's say I'm looking at a forest plot from a meta-analysis. I notice that most of the width of the confidence intervals are fairly consistent and the point estimates are all on one side (showing a ...
- 31
0
votes
1
answer
244
views
Analyzing replicates of contingency tables
Problem statement
We would like to measure the effect of treatment on cell differentiation, in other words whether or not specific treatments cause cells to change their cell types. So we treat cells, ...
- 25
1
vote
0
answers
47
views
Two-Way ANOVA with significant interaction with replication, how to proceed?
I am performing an omnibus two-way ANOVA test. In the set of factors that I want to test I included biological replication and technical replication.
The function used for the test was:
...
- 21
1
vote
1
answer
578
views
Should I average data sets and calculate single parameter or average individual data set's parameters?
I am interested in calculating oxidation reaction kinetics parameters of a material at a particular temperature. Essentially I have a curve of data (mass change) vs. time for the particular ...
- 11
10
votes
2
answers
307
views
Automated ML vs the entire replicability/reproducibility crisis
There is a trend in machine learning implementations to make things easier and easier for implementers, a very natural engineering concern. Easy APIs to create any kind of model you want, easy ...
- 1,843
1
vote
0
answers
43
views
Replicating a wrong analysis
A study measures the performance of patients and controls in two tasks.
In task 1, a t-test reveals no significant differences between patients and controls. In task 2, a t-test reveals significant ...
- 381
4
votes
1
answer
162
views
Why replication studies use two-tailed tests?
I have been searching replication studies and I found that all of them use two-tailed tests. For example:
Open Science Collaboration. (2015). Estimating the reproducibility of psychological science....
- 381
7
votes
0
answers
72
views
Has the reproducibility crisis affected confidence intervals as well?
The reproducibility crisis has given many pause over the value (?) of $p$-values to measure the relevance of statistical findings. Given the interpretation of a $p$-value and some knowledge of ...
- 57.9k
1
vote
0
answers
98
views
Does this analysis commit pseudoreplication?
I designed my experiment in a way that I thought would present me with options for analysis, but I'm starting to wonder if one of the options would be considered pseudoreplication.
The experiment was ...
- 151
0
votes
2
answers
55
views
Within-sample p-value VS across-samples p-value for the same test statistic
I have a volume of brain which contains thousands of connections between hundreds of cells. Suppose I observe that Cell type A connects three times more to Cell type B than Cell type C, and calculate ...
- 13
0
votes
2
answers
87
views
replicability and statistical significance
Data for two nearly identical clinical trials. Study A had 750 pts, 374 placebo and 376 drug. Study B had 752 pts, 376 placebo and 376 drug. Same protocol for both. Study A showed mean change from ...
- 21
3
votes
0
answers
202
views
Repeating experiment with biological replicates
My experiment gives data of 12 biological replicates. This would be sufficient to calculate the mean and give a valid standard deviation. However, there might have been some pipetting errors that ...
- 151
3
votes
2
answers
6k
views
Biological and technical replicates for statistical analysis in cellular biology
These are questions regarding basic statistics/reporting in biology. I have already read a couple of articles on this subject, but couldn't find a clear answer applying to my research.
I have the ...
- 33
8
votes
1
answer
190
views
Meaning of low power in neuroscience after combining results of many meta-analyses (Button et al 2013)
In a 2013 review article in Nature Neuroscience, Button et al. Power failure: why small sample size undermines the reliability of neuroscience, it was stated that:
the average statistical power of ...
- 735
3
votes
2
answers
97
views
Why is the power of studies that only report significant effects not always 100%?
As I was reading the following passage of this blog, which defines R-(replicability-)indices:
To correct for the inflation in power, the R-Index uses the inflation
rate. For example, if all ...
- 2,150
1
vote
0
answers
48
views
Replication after Meta-Analysis in a multiple testing situation?
I was recently in an introductory seminar about Genome Wide Association Studies (GWAS). These studies aim to examine if any genetic markers are associate with a certain trait. It is not uncommon to ...
- 221
52
votes
4
answers
1k
views
Cumming (2008) claims that distribution of p-values obtained in replications depends only on the original p-value. How can it be true?
I have been reading Geoff Cumming's 2008 paper Replication and $p$ Intervals: $p$ values predict the future only vaguely, but confidence intervals do much better [~200 citations in Google Scholar] -- ...
- 100k
1
vote
1
answer
193
views
Replication study and Bayes factor
I ran an experiment and got a statistically significant result with an effect size of cohen's d=0.36 (N=20). I decided to run a replication study, this time with 60 participants, and got nothing (d=0....
- 425
13
votes
2
answers
746
views
What fraction of repeat experiments will have an effect size within the 95% confidence interval of the first experiment?
Let's stick to an ideal situation with random sampling, Gaussian populations, equal variances, no P-hacking, etc.
Step 1. You run an experiment say comparing two sample means, and compute a 95% ...
- 18.4k
1
vote
0
answers
1k
views
Pooling data from replicate experiments 2
I have a problem regarding pooling data from replicate experiments.
I have done an experiment where I collected multiple tissue samples from 5 mice, then the samples were allocated to treatments and ...
- 41
1
vote
1
answer
49
views
Evidence for data mining (specification search) in published results [duplicate]
Is there an established way to assess the prevalence of data mining (as in specification search, not in the machine learning sense) in academic publications?
I vaguely remember hearing something ...
- 337
1
vote
0
answers
25
views
Testing two groups of subjects (same population) -> increased replicability?
I've seen several experimental psychology papers arguing that, because they separately tested two cohorts of subjects, that this increases the replicability of their finding. Is this really the case? ...
- 2,150
2
votes
1
answer
132
views
How to identify studies that should be replicated?
In psychology voting on which studies should be replicated is established on a website. The ReplicationWiki (that I founded) offers a voting option for studies in economics and related fields, but it ...
- 171
1
vote
1
answer
2k
views
Pooling data from replicate experiments
Let's say we are asking if a vaccine is effective in mice. We test the vaccine vs control in groups of 10 mice, and the vaccine does confer a significant immune response (based on a linear mixed ...
- 107
6
votes
1
answer
540
views
Independent replication experiments yielding contrasting results; how to combine them?
Imagine a simple experiment, trying to answer a simple question. For example, is body temperature the same in men and in women ?
To answer this question, let's say you sample 10 men, and 10 women, ...
- 1,023
4
votes
1
answer
60
views
If original data are available, should replication data be added and everything re-analysed (IPD) or is a meta-analysis better?
One common practice which increases the odds of obtaining spurious
results is to keep collecting observations after a preliminary
analyses are performed[1]. This occurs when the cutoff point for
...
- 697
7
votes
3
answers
332
views
Study replication from a Bayesian point of view
Consider following situation: Study A compares two groups and finds a mean difference with an effect size of d = .8 (p<.05). Study B is a direct replication, and finds an effect in the same ...
- 4,620