Questions tagged [replicability]

The extent to which subsequent studies of the same phenomenon reproduce the results obtained in the original study.

1
vote
1answer
33 views

Should I average data sets and calculate single parameter or average individual data set's parameters?

I am interested in calculating oxidation reaction kinetics parameters of a material at a particular temperature. Essentially I have a curve of data (mass change) vs. time for the particular ...
0
votes
0answers
47 views

When conducting a study, what is standard practice with patterns you find along the way? [closed]

In olden times, say, five years ago, it was common practice to collect a bunch of data, do statistics on them, and present it all emphasizing the 'statistically significant' items that support your ...
0
votes
0answers
20 views

Troubles with Basket-Sensitive Recommendation System using CF + Modified Random Walk

I'm trying to reproduce and understand the "Basket-Sensitive Random Walk" for recommendation systems proposed in the paper "Grocery Shopping Recommendations Based on Basket-Sensitive Random Walk" of ...
6
votes
0answers
62 views

Automated ML vs the entire replicability/reproducibility crisis

There is a trend in machine learning implementations to make things easier and easier for implementers, a very natural engineering concern. Easy APIs to create any kind of model you want, easy ...
1
vote
0answers
37 views

Replicating a wrong analysis

A study measures the performance of patients and controls in two tasks. In task 1, a t-test reveals no significant differences between patients and controls. In task 2, a t-test reveals significant ...
2
votes
1answer
42 views

Why replication studies use two-tailed tests?

I have been searching replication studies and I found that all of them use two-tailed tests. For example: Open Science Collaboration. (2015). Estimating the reproducibility of psychological science....
0
votes
0answers
20 views

Confidence intervals interpretation in whole population studies. Can they represent replication variability?

I got the surgery time of each surgery ward patient. I want to compare the mean surgery time of each ward using a series of regressions in which the independent variable is being or not in a specific ...
1
vote
0answers
34 views

Does this analysis commit pseudoreplication?

I designed my experiment in a way that I thought would present me with options for analysis, but I'm starting to wonder if one of the options would be considered pseudoreplication. The experiment was ...
0
votes
2answers
41 views

Within-sample p-value VS across-samples p-value for the same test statistic

I have a volume of brain which contains thousands of connections between hundreds of cells. Suppose I observe that Cell type A connects three times more to Cell type B than Cell type C, and calculate ...
0
votes
2answers
75 views

replicability and statistical significance

Data for two nearly identical clinical trials. Study A had 750 pts, 374 placebo and 376 drug. Study B had 752 pts, 376 placebo and 376 drug. Same protocol for both. Study A showed mean change from ...
3
votes
0answers
92 views

Repeating experiment with biological replicates

My experiment gives data of 12 biological replicates. This would be sufficient to calculate the mean and give a valid standard deviation. However, there might have been some pipetting errors that ...
2
votes
2answers
3k views

Biological and technical replicates for statistical analysis in cellular biology

These are questions regarding basic statistics/reporting in biology. I have already read a couple of articles on this subject, but couldn't find a clear answer applying to my research. I have the ...
7
votes
1answer
162 views

Meaning of low power in neuroscience after combining results of many meta-analyses (Button et al 2013)

In a 2013 review article in Nature Neuroscience, Button et al. Power failure: why small sample size undermines the reliability of neuroscience, it was stated that: the average statistical power of ...
2
votes
2answers
62 views

Why is the power of studies that only report significant effects not always 100%?

As I was reading the following passage of this blog, which defines R-(replicability-)indices: To correct for the inflation in power, the R-Index uses the inflation rate. For example, if all ...
1
vote
0answers
38 views

Replication after Meta-Analysis in a multiple testing situation?

I was recently in an introductory seminar about Genome Wide Association Studies (GWAS). These studies aim to examine if any genetic markers are associate with a certain trait. It is not uncommon to ...
52
votes
4answers
1k views

Cumming (2008) claims that distribution of p-values obtained in replications depends only on the original p-value. How can it be true?

I have been reading Geoff Cumming's 2008 paper Replication and $p$ Intervals: $p$ values predict the future only vaguely, but confidence intervals do much better [~200 citations in Google Scholar] -- ...
0
votes
1answer
167 views

Replication study and Bayes factor

I ran an experiment and got a statistically significant result with an effect size of cohen's d=0.36 (N=20). I decided to run a replication study, this time with 60 participants, and got nothing (d=0....
12
votes
2answers
263 views

What fraction of repeat experiments will have an effect size within the 95% confidence interval of the first experiment?

Let's stick to an ideal situation with random sampling, Gaussian populations, equal variances, no P-hacking, etc. Step 1. You run an experiment say comparing two sample means, and compute a 95% ...
1
vote
0answers
472 views

Pooling data from replicate experiments 2

I have a problem regarding pooling data from replicate experiments. I have done an experiment where I collected multiple tissue samples from 5 mice, then the samples were allocated to treatments and ...
1
vote
1answer
43 views

Evidence for data mining (specification search) in published results [duplicate]

Is there an established way to assess the prevalence of data mining (as in specification search, not in the machine learning sense) in academic publications? I vaguely remember hearing something ...
1
vote
0answers
22 views

Testing two groups of subjects (same population) -> increased replicability?

I've seen several experimental psychology papers arguing that, because they separately tested two cohorts of subjects, that this increases the replicability of their finding. Is this really the case? ...
2
votes
1answer
114 views

How to identify studies that should be replicated?

In psychology voting on which studies should be replicated is established on a website. The ReplicationWiki (that I founded) offers a voting option for studies in economics and related fields, but it ...
1
vote
1answer
886 views

Pooling data from replicate experiments

Let's say we are asking if a vaccine is effective in mice. We test the vaccine vs control in groups of 10 mice, and the vaccine does confer a significant immune response (based on a linear mixed ...
5
votes
1answer
384 views

Independent replication experiments yielding contrasting results; how to combine them?

Imagine a simple experiment, trying to answer a simple question. For example, is body temperature the same in men and in women ? To answer this question, let's say you sample 10 men, and 10 women, ...
3
votes
1answer
55 views

If original data are available, should replication data be added and everything re-analysed (IPD) or is a meta-analysis better?

One common practice which increases the odds of obtaining spurious results is to keep collecting observations after a preliminary analyses are performed[1]. This occurs when the cutoff point for ...
7
votes
3answers
295 views

Study replication from a Bayesian point of view

Consider following situation: Study A compares two groups and finds a mean difference with an effect size of d = .8 (p<.05). Study B is a direct replication, and finds an effect in the same ...