Skip to main content

Questions tagged [reproducible-research]

Research practice of making full experimental description, entire collected data, and all the data analysis scripts publicly available, so that the published results can be reproduced elsewhere.

Filter by
Sorted by
Tagged with
0 votes
1 answer
31 views

Recreate `lm` Categorical Regression

Consider the code, which contains regression using lm of two categorical and one continuous variables without interaction using data from the correct model: ...
温泽海's user avatar
  • 456
0 votes
1 answer
45 views

Power of One Sided t-test

Let $X_1, \ldots, X_n$ be a sample from $N(\mu, \sigma^2)$ for unknown $\mu \in \mathbb{R}$ and unknown $\sigma > 0$. Fix $\mu_0 \in \mathbb{R}$. The one-sided hypothesis is $H_0: \mu \leqslant \...
温泽海's user avatar
  • 456
3 votes
2 answers
149 views

Mathematical Prediction of Linear Mixed Models Random Intercept

Given data $\{(x_{i,j}, y_{i,j})\} \subset \mathbb{R}^2$, with $i = 1, \ldots, k$ classes and $j = 1, \ldots, n_i$. The linear mixed model is: \begin{equation*} y_{i,j} = a + b x_{i,j} + u_i + \...
温泽海's user avatar
  • 456
1 vote
1 answer
58 views

Interpreting variable selection performance on N datasets using N glmnet LASSOs run with the same lambda of s = 0.1 vs cv.glmnet with s = lambda.1se

This question is a follow-up to this prior question on here from a week ago which was itself a follow-up to this prior question. Here is a link to the GitHub Repository for the project. I am running N ...
Marlen's user avatar
  • 147
1 vote
1 answer
139 views

Why are the performance of LASSOs implemented via glmnet in R going down after cross-validation?

This question is a follow-up to this question I asked here last week. I got an important and useful answer to it, but what that led to was surprising. And once again, here is a link to the GitHub Repo ...
Marlen's user avatar
  • 147
1 vote
1 answer
120 views

Different sets of features selected by three different functions in R for running LASSO Regressions despite the same random seed for each

The GitHub Repository for this research project has all of the code included in this question. Brief background context: I am just finishing up the work on my part as a coauthor on a research project ...
Marlen's user avatar
  • 147
4 votes
1 answer
82 views

What is the current status of Killeen’s probability of replication?

Killeen (1) published several papers about P-rep. It can be easily computed from a p-value (for a two-group comparison) and has an easy interpretation. It is the estimated probability that future ...
Harvey Motulsky's user avatar
5 votes
2 answers
299 views

If p < 0.05 in one experiment, what is the probability of p < 0.05 in a repeat experiment?

With all the concern about reproducibility, I have not seen a very basic question answered. Using the standard hypothesis testing approach, if one experiment results in p<0.05, what is the chance ...
Harvey Motulsky's user avatar
3 votes
1 answer
60 views

Can't reproduce closed form expression from sequential testing

I've been trying to implement and extend some results from the papers "Always Valid Inference" and "Peeking at A/B Tests". The authors provide a closed form expression of the "...
user2844936's user avatar
1 vote
1 answer
33 views

Does a newly constructed ML dataset need to have an official train-dev-test split? Should the test set be intentionally balanced?

I have constructed a novel ML (NLP) dataset for classification and labeled it with three classes. The dataset is rather small with about 700 examples, out of which the classes have about 400, 200, and ...
Arno's user avatar
  • 11
4 votes
1 answer
225 views

Fine-tuning of methods before main analysis: p-value hacking?

I am reviewing a paper for a journal. The authors propose a new classification scheme. They then try to assess the classification accuracy using some novel predictor data. Standard classification ...
user7831861's user avatar
0 votes
1 answer
78 views

How can Mass Spectrometry have high sensitivity but average reproducibility?

I am reading up on mass spec from this link: https://www.ebi.ac.uk/training/online/courses/metabolomics-introduction/designing-a-metabolomics-study/comparison-of-nmr-and-ms/ They state: The two most ...
statHacker's user avatar
1 vote
0 answers
102 views

Python implementation of standard Expectation maximization algorithm to estimate position bias in LTR

Is there any Python implementation that would allow to use logs data of clicks and their ranking, and return and estimate of relative position bias as discussed in section ...
Tony's user avatar
  • 11
1 vote
0 answers
90 views

How to validate a clustering model on new data?

I was wondering how one would properly validate the reproducibility of a trained clustering model on a completely new set of data. Imagine you are clustering a patient population in hospital A. You ...
The Jipsess's user avatar
1 vote
0 answers
34 views

How to ensure repeatability and reproducibility in a clinical study?

I have a 2 arm clinical study with topical anesthetic study group and placebo on the two arms. I measured the pain scores of patients with a pain scale. It is not an experimental study.In the method ...
karaca's user avatar
  • 71
1 vote
0 answers
94 views

Can I use both a Kolmogorov–Smirnov test and a T-test to compare two distributions?

I have two distributions (N ~ 300, μ = 1.89 and σ = 0.97) and (N ~ 40, μ = 1.74 and σ = 1.18). A ks_2samp returns statistic = 0.2 and p-value = 0.057. Therefore, I cannot reject the null hypothesis ...
José Carlo's user avatar
3 votes
2 answers
129 views

Could we estimate replicability of empirical research with conformal predictions?

A review article Threats of a Replication Crisis in Empirical Computer Science reviews reproducibility issues. The authors present distinctions among repeatability, replicability, and reproducibility. ...
patagonicus's user avatar
  • 2,550
0 votes
0 answers
467 views

Help interpreting event study methodology of a famous research article

I wonder if someone out there familiar with reading research articles, especially finance research can help me interpret which event study methodology the author uses in her famous research article. ...
thomas.diridondo's user avatar
0 votes
1 answer
79 views

Gauge R&R analysis - Preicision of a system

I never attempted to do a GR&R analysis before so I'm sharing an overview of what I would like to attempt to do and hopefully I would get some insight/advice/reference on which approach I should ...
the phoenix's user avatar
2 votes
0 answers
57 views

Understanding a simple post-hoc power analysis of effect size

I am trying to understand a relatively straightforward paragraph found in "Gun Shows and Gun Violence: Fatally Flawed Study Yields Misleading Results" by Wintermute et al that appeared in ...
bashity's user avatar
  • 121
2 votes
1 answer
497 views

Help in re-producing the calculation of the standardised difference for a multinomial variable

I am interested in calculating the standardised difference for a multinomial variable. In R there is a package called tableone that produces such calculations (https://cran.r-project.org/web/packages/...
Stephen Clark's user avatar
1 vote
2 answers
678 views

Random number generator seed in R

I have a question about the random number generator seed number in R. Recently, I am trying to solve the exercises in the book named "An Introduction to Statistical Learning". When I ...
M.C. Park's user avatar
  • 935
0 votes
1 answer
53 views

Best Practices in Sampling Large Datasets

I am working on a research paper concerning developing a CNN model for a multi-class classification on images. I have a large dataset consisting of 3 classes summing up to 20000 images. Class 1 has ...
djbacs's user avatar
  • 51
3 votes
0 answers
40 views

To what degree does reproducibility of research differ across disciplines?

Have there been any statistics meta studies that compare the reproducibility of results, as well as the willingness of researchers to provide reproducible data, analysis, and code, in their studies ...
stevec's user avatar
  • 303
2 votes
1 answer
3k views

Data science project examples based on Cookie Cutter Data science template [closed]

Interested by production ready data science project do you have any public github reference concrete implementation? As an example i found this project template structure https://drivendata.github.io/...
amous's user avatar
  • 51
1 vote
1 answer
86 views

What is the right machine learning research methodology to compare two neural network approaches against eachother?

I am new to machine learning research, and I have a general question regarding how to compare different models on the same data set. How does a top researcher do this, and what is expected from a ...
user3352632's user avatar
4 votes
1 answer
303 views

Ablation Study training until when?

Given that deep learning models can take weeks to train, having to train 20 different models becomes infeasible when performing an ablation study. Are there some formally accepted common practices on ...
iordanis's user avatar
  • 485
0 votes
1 answer
463 views

Predict values from complex Rjags model

It's the first time I'm working with R2Jags, MCM chains and Bayesian models and I'm having trouble to compute the predicted values for my model. The model is based on research by Hallmann et al. 2017, ...
L.Thoma's user avatar
1 vote
0 answers
20 views

Reproducibility Between Remote and Local Computer [closed]

I am currently using R, RStudio, and a remote computer (using an R script) to run the same ...
AW27's user avatar
  • 253
3 votes
1 answer
45 views

Appropriate model for amount of statistical errors in articles

I recently started my PhD and I am currently working on a project about finding statistical reporting errors. Our work is similar to Nuijten et al. (2016) only for economics. So, I have a database ...
Irazall's user avatar
  • 170
0 votes
0 answers
271 views

Failing to replicate the signal-to-noise ratio from original lasso paper

I'm referencing example 1 under the simulations section from Tibshirani. In short there's a model in the form $$ y = \beta^Tx + \sigma \epsilon, $$ where $\beta = (3, 1.5, 0, 0, 2, 0, 0, 0)^T$ and $\...
GintsB's user avatar
  • 1
0 votes
1 answer
480 views

Random number generator independence

I have a naïve question about random number generators. Say I want to generate a set of random variables from different exponential distributions,$$X_i \sim \mathcal {Exp}(\lambda_i)$$ Assume the$X_i$'...
Jeffrey's user avatar
  • 107
1 vote
1 answer
247 views

Pearson's Correlation between samples after PCA dimensionality reduction

This question is closely related to [Is it acceptable to reverse a sign of a principal component score?. I have a matrix A with Samples in row and Features in column. I apply PCA on A and get a ...
Paquito's user avatar
  • 11
0 votes
0 answers
25 views

Research on neural networks mimicking behavior

My question is regarding neural networks mimicking noise specifically. The idea. I have clinically clean image generated by a ray tracer and a neural network (type does not matter) that is mimicking ...
Marko Bencik's user avatar
1 vote
0 answers
44 views

How to determine statistical significance of physics dataset?

I am working on a physics paper, which looks at the dependence of resistivity on physical temperature. I obtained data showing a linear trend of the form $$\rho(T)=\rho_0[1+\alpha(T-T_0)]$$ Based on ...
BooleanDesigns's user avatar
0 votes
0 answers
38 views

Setting seeds despite repeated training of CNNs?

I would like to compare the classification performance (like accuracy, precision, recall etc.) of different CNN architectures. I'm using Google Colab (GPU support), Tensorflow and Keras. Since it is ...
Code Now's user avatar
0 votes
1 answer
72 views

How to compare the performance of different SVMs and CNNs?

I'm a beginner in machine learning und I have a problem to find the best way to compare the performance (accuracy) of different SVMs and CNNs (Jupyter Notebook)? The CNNs I train in Google Colab with ...
Code Now's user avatar
0 votes
1 answer
1k views

How to compare CNN models with non-reproducible results?

I try to compare different CNN models. I use Keras and for training, I use a GPU, Google Colab with Tensorflow backend. Unfortunately I'm not able to create the same initial conditions for the CNNs (...
Code Now's user avatar
0 votes
0 answers
185 views

Does a high negative concordance imply reproducible features

Does a high negative(-1) Lin's concordance correlation coefficient(Lins CCC) imply reproducible features ? I am calculating Lin's CCC of radiomic features between two different CT scan settings , ...
Michael Moran's user avatar
2 votes
1 answer
234 views

Is most published research indeed false?

I have come across (Ioannidis, 2005) which explains several reasons (mainly statistics-related, that's why I post this question here) to justify the claim that most published research is indeed false. ...
David's user avatar
  • 2,665
2 votes
1 answer
298 views

Reproducible numbers in Keras/TensorFlow

Every time I run a Keras/TensorFlow code gives different results. Can someone suggest how to get reproducible numbers?
talk2speech's user avatar
9 votes
1 answer
787 views

Why do people use PCA when it has so many issues?

(This is a soft question) Recently I'm learning Principal Component Analysis, and it appears to have a lot of issues: You have to transform the data to roughly the same scale before applying PCA, but ...
nalzok's user avatar
  • 1,797
1 vote
0 answers
43 views

Replicating a wrong analysis

A study measures the performance of patients and controls in two tasks. In task 1, a t-test reveals no significant differences between patients and controls. In task 2, a t-test reveals significant ...
danilinares's user avatar
4 votes
1 answer
203 views

Why replication studies use two-tailed tests?

I have been searching replication studies and I found that all of them use two-tailed tests. For example: Open Science Collaboration. (2015). Estimating the reproducibility of psychological science....
danilinares's user avatar
7 votes
0 answers
75 views

Has the reproducibility crisis affected confidence intervals as well?

The reproducibility crisis has given many pause over the value (?) of $p$-values to measure the relevance of statistical findings. Given the interpretation of a $p$-value and some knowledge of ...
AdamO's user avatar
  • 63.8k
0 votes
1 answer
254 views

What is a good machine learning academic research workflow/tools? [closed]

I'm interested in ML research and want to get some insights into how its done in practice. In particular: What kind of data are you dealing with? How are you dealing with hyper-param optimization? ...
matan tsuberi's user avatar
0 votes
0 answers
33 views

Use Deep learning or RL to maximize long term hit in caching file?

Suppose I have F video files with different file size and with different popularity. I have a memory cache of fixed size. What I have to do is following: I want to cache n files (n < F) in ...
ironman's user avatar
  • 702
6 votes
2 answers
12k views

Choosing the "Correct" Seed for Reproducible Research/Results

I have created this post to spark a discussion around an idea I had regarding the choice of the seed number used to replicate the results of a statistical model. Here is some background on how I came ...
Brandon's user avatar
  • 71
2 votes
0 answers
83 views

Put Together Results from GLM Gamma Models

I have a set of healthcare data in which I used GLM Gamma to model the healthcare spending. Then I am trying to put together the results in a style that is similar to this: http://onlinelibrary.wiley....
user2935184's user avatar
6 votes
1 answer
683 views

"report the statistical results of the analysis without the covariate"

This is from Psychological Science 22(11), p.1363: If an analysis includes a covariate, authors must report the statistical results of the analysis without the covariate. Reporting covariate-...
user66081's user avatar
  • 171