Skip to main content

Questions tagged [replicability]

The extent to which subsequent studies of the same phenomenon reproduce the results obtained in the original study.

Filter by
Sorted by
Tagged with
2 votes
2 answers
198 views

For normality testing, should biological replicates be averaged or should all values of each subgroup be evaluated as independent data points?

I have performed an in vivo study evaluating splenic white:red pulp ratio as determined by AI on histopathologic slides (neural network trained and validated per published guidelines). The questions ...
NCZ's user avatar
  • 31
6 votes
2 answers
184 views

What is the rationale for computing a confidence interval in a study that won't or can't be repeated?

As far as I understand it, when we compute a confidence interval (say a $95$% CI), we have confidence in the procedure, not in the current data we have (as a $95$% CI means that $95$% of confidence ...
user406426's user avatar
2 votes
1 answer
61 views

How to Replicate result that probabilities in included table are different 7 times?

Question: In the table at the bottom of this question, the total number of times that $\hat{P}(x;z) > \hat{P}_y(x;z) $ and $\hat{P}(y;z)>\hat{P}_x(y;z)$ is 7. Can someone please replicate this ...
user106860's user avatar
1 vote
1 answer
52 views

Experiment design useful when objects change / will be replaced in each trial run?

I want to perform some studies on how material thickness, type of material, humidity and temperature influence tensile strength. While I was thinking about useful designs I was wondering whether a ...
Ben's user avatar
  • 3,473
0 votes
1 answer
72 views

What are tests to determine the number of technical replicates neccessary to reach the true mean?

I have the following scenario: I am working with a quite variable material, so I am using technical replicates to estimate the mean sample properties. To test how many replicas I need to converge to ...
Dunen's user avatar
  • 175
3 votes
2 answers
126 views

Could we estimate replicability of empirical research with conformal predictions?

A review article Threats of a Replication Crisis in Empirical Computer Science reviews reproducibility issues. The authors present distinctions among repeatability, replicability, and reproducibility. ...
patagonicus's user avatar
  • 2,540
2 votes
1 answer
62 views

Can confidence intervals or uncertainty intervals provide information on replication? [duplicate]

Let's say I'm looking at a forest plot from a meta-analysis. I notice that most of the width of the confidence intervals are fairly consistent and the point estimates are all on one side (showing a ...
Dylan A's user avatar
  • 31
0 votes
1 answer
365 views

Analyzing replicates of contingency tables

Problem statement We would like to measure the effect of treatment on cell differentiation, in other words whether or not specific treatments cause cells to change their cell types. So we treat cells, ...
perlusha's user avatar
1 vote
0 answers
48 views

Two-Way ANOVA with significant interaction with replication, how to proceed?

I am performing an omnibus two-way ANOVA test. In the set of factors that I want to test I included biological replication and technical replication. The function used for the test was: ...
Rosario's user avatar
  • 21
1 vote
1 answer
1k views

Should I average data sets and calculate single parameter or average individual data set's parameters?

I am interested in calculating oxidation reaction kinetics parameters of a material at a particular temperature. Essentially I have a curve of data (mass change) vs. time for the particular ...
D. Hallatt's user avatar
11 votes
2 answers
322 views

Automated ML vs the entire replicability/reproducibility crisis

There is a trend in machine learning implementations to make things easier and easier for implementers, a very natural engineering concern. Easy APIs to create any kind of model you want, easy ...
Mitch's user avatar
  • 1,987
1 vote
0 answers
43 views

Replicating a wrong analysis

A study measures the performance of patients and controls in two tasks. In task 1, a t-test reveals no significant differences between patients and controls. In task 2, a t-test reveals significant ...
danilinares's user avatar
4 votes
1 answer
197 views

Why replication studies use two-tailed tests?

I have been searching replication studies and I found that all of them use two-tailed tests. For example: Open Science Collaboration. (2015). Estimating the reproducibility of psychological science....
danilinares's user avatar
7 votes
0 answers
75 views

Has the reproducibility crisis affected confidence intervals as well?

The reproducibility crisis has given many pause over the value (?) of $p$-values to measure the relevance of statistical findings. Given the interpretation of a $p$-value and some knowledge of ...
AdamO's user avatar
  • 63k
1 vote
0 answers
117 views

Does this analysis commit pseudoreplication?

I designed my experiment in a way that I thought would present me with options for analysis, but I'm starting to wonder if one of the options would be considered pseudoreplication. The experiment was ...
Nathan Haag's user avatar
0 votes
2 answers
57 views

Within-sample p-value VS across-samples p-value for the same test statistic

I have a volume of brain which contains thousands of connections between hundreds of cells. Suppose I observe that Cell type A connects three times more to Cell type B than Cell type C, and calculate ...
Alex's user avatar
  • 13
0 votes
2 answers
88 views

replicability and statistical significance

Data for two nearly identical clinical trials. Study A had 750 pts, 374 placebo and 376 drug. Study B had 752 pts, 376 placebo and 376 drug. Same protocol for both. Study A showed mean change from ...
Tom G's user avatar
  • 21
3 votes
0 answers
233 views

Repeating experiment with biological replicates

My experiment gives data of 12 biological replicates. This would be sufficient to calculate the mean and give a valid standard deviation. However, there might have been some pipetting errors that ...
SecondLemon's user avatar
3 votes
2 answers
6k views

Biological and technical replicates for statistical analysis in cellular biology

These are questions regarding basic statistics/reporting in biology. I have already read a couple of articles on this subject, but couldn't find a clear answer applying to my research. I have the ...
Dunn Wallis's user avatar
8 votes
1 answer
195 views

Meaning of low power in neuroscience after combining results of many meta-analyses (Button et al 2013)

In a 2013 review article in Nature Neuroscience, Button et al. Power failure: why small sample size undermines the reliability of neuroscience, it was stated that: the average statistical power of ...
arkiaamu's user avatar
  • 765
3 votes
2 answers
155 views

Why is the power of studies that only report significant effects not always 100%?

As I was reading the following passage of this blog, which defines R-(replicability-)indices: To correct for the inflation in power, the R-Index uses the inflation rate. For example, if all ...
z8080's user avatar
  • 2,422
1 vote
0 answers
51 views

Replication after Meta-Analysis in a multiple testing situation?

I was recently in an introductory seminar about Genome Wide Association Studies (GWAS). These studies aim to examine if any genetic markers are associate with a certain trait. It is not uncommon to ...
DUWUDA's user avatar
  • 231
52 votes
4 answers
2k views

Cumming (2008) claims that distribution of p-values obtained in replications depends only on the original p-value. How can it be true?

I have been reading Geoff Cumming's 2008 paper Replication and $p$ Intervals: $p$ values predict the future only vaguely, but confidence intervals do much better [~200 citations in Google Scholar] -- ...
amoeba's user avatar
  • 105k
1 vote
1 answer
198 views

Replication study and Bayes factor

I ran an experiment and got a statistically significant result with an effect size of cohen's d=0.36 (N=20). I decided to run a replication study, this time with 60 participants, and got nothing (d=0....
TanZor's user avatar
  • 425
13 votes
2 answers
787 views

What fraction of repeat experiments will have an effect size within the 95% confidence interval of the first experiment?

Let's stick to an ideal situation with random sampling, Gaussian populations, equal variances, no P-hacking, etc. Step 1. You run an experiment say comparing two sample means, and compute a 95% ...
Harvey Motulsky's user avatar
1 vote
0 answers
1k views

Pooling data from replicate experiments 2

I have a problem regarding pooling data from replicate experiments. I have done an experiment where I collected multiple tissue samples from 5 mice, then the samples were allocated to treatments and ...
Rebecca's user avatar
  • 41
1 vote
1 answer
52 views

Evidence for data mining (specification search) in published results [duplicate]

Is there an established way to assess the prevalence of data mining (as in specification search, not in the machine learning sense) in academic publications? I vaguely remember hearing something ...
sheß's user avatar
  • 367
1 vote
0 answers
26 views

Testing two groups of subjects (same population) -> increased replicability?

I've seen several experimental psychology papers arguing that, because they separately tested two cohorts of subjects, that this increases the replicability of their finding. Is this really the case? ...
z8080's user avatar
  • 2,422
4 votes
1 answer
144 views

How to identify studies that should be replicated?

In psychology voting on which studies should be replicated is established on a website. The ReplicationWiki (that I founded) offers a voting option for studies in economics and related fields, but it ...
Jan Höffler's user avatar
1 vote
1 answer
2k views

Pooling data from replicate experiments

Let's say we are asking if a vaccine is effective in mice. We test the vaccine vs control in groups of 10 mice, and the vaccine does confer a significant immune response (based on a linear mixed ...
iayork's user avatar
  • 107
6 votes
1 answer
576 views

Independent replication experiments yielding contrasting results; how to combine them?

Imagine a simple experiment, trying to answer a simple question. For example, is body temperature the same in men and in women ? To answer this question, let's say you sample 10 men, and 10 women, ...
Rodolphe's user avatar
  • 1,033
4 votes
1 answer
63 views

If original data are available, should replication data be added and everything re-analysed (IPD) or is a meta-analysis better?

One common practice which increases the odds of obtaining spurious results is to keep collecting observations after a preliminary analyses are performed[1]. This occurs when the cutoff point for ...
post-hoc's user avatar
  • 697
7 votes
3 answers
339 views

Study replication from a Bayesian point of view

Consider following situation: Study A compares two groups and finds a mean difference with an effect size of d = .8 (p<.05). Study B is a direct replication, and finds an effect in the same ...
Felix S's user avatar
  • 4,700