Questions tagged [reproducible-research]

Research practice of making full experimental description, entire collected data, and all the data analysis scripts publicly available, so that the published results can be reproduced elsewhere.

Filter by
Sorted by
Tagged with
1
vote
1answer
15 views

Help in re-producing the calculation of the standardised difference for a multinomial variable

I am interested in calculating the standardised difference for a multinomial variable. In R there is a package called tableone that produces such calculations (https://cran.r-project.org/web/packages/...
1
vote
2answers
80 views

Random number generator seed in R

I have a question about the random number generator seed number in R. Recently, I am trying to solve the exercises in the book named "An Introduction to Statistical Learning". When I ...
0
votes
1answer
14 views

Best Practices in Sampling Large Datasets

I am working on a research paper concerning developing a CNN model for a multi-class classification on images. I have a large dataset consisting of 3 classes summing up to 20000 images. Class 1 has ...
3
votes
0answers
25 views

To what degree does reproducibility of research differ across disciplines?

Have there been any statistics meta studies that compare the reproducibility of results, as well as the willingness of researchers to provide reproducible data, analysis, and code, in their studies ...
1
vote
1answer
158 views

Data science project examples based on Cookie Cutter Data science template [closed]

Interested by production ready data science project do you have any public github reference concrete implementation? As an example i found this project template structure https://drivendata.github.io/...
0
votes
1answer
33 views

What is the right machine learning research methodology to compare two neural network approaches against eachother?

I am new to machine learning research, and I have a general question regarding how to compare different models on the same data set. How does a top researcher do this, and what is expected from a ...
1
vote
0answers
22 views

Ablation Study training until when?

Given that deep learning models can take weeks to train, having to train 20 different models becomes infeasible when performing an ablation study. Are there some formally accepted common practices on ...
0
votes
1answer
25 views

Predict values from complex Rjags model

It's the first time I'm working with R2Jags, MCM chains and Bayesian models and I'm having trouble to compute the predicted values for my model. The model is based on research by Hallmann et al. 2017, ...
0
votes
0answers
22 views

Calculating significance on incidence rates

This is my first time attempting a project like this so my question might be quite wordy. I am analysing the trends in incidence of disease X on the population over 30 years. I have grouped the 30 ...
0
votes
0answers
7 views

Can I compare two different algorithms one with early stopping and one without early stopping?

I develop an algorithm based on action recognition task. Now I want to compare the performance of my algorithm with another algorithm. But the other algorithm that I am interested in comparing uses ...
0
votes
0answers
526 views

A critical assumption of the exploratory factor analysis (EFA)

A critical assumption of the exploratory factor analysis (EFA) is that it is only appropriate for sets of non-nominal items which theoretically belong to reflective latent factors. Why is it so ...
1
vote
0answers
19 views

Reproducibility Between Remote and Local Computer [closed]

I am currently using R, RStudio, and a remote computer (using an R script) to run the same ...
3
votes
1answer
40 views

Appropriate model for amount of statistical errors in articles

I recently started my PhD and I am currently working on a project about finding statistical reporting errors. Our work is similar to Nuijten et al. (2016) only for economics. So, I have a database ...
0
votes
0answers
61 views

Failing to replicate the signal-to-noise ratio from original lasso paper

I'm referencing example 1 under the simulations section from Tibshirani. In short there's a model in the form $$ y = \beta^Tx + \sigma \epsilon, $$ where $\beta = (3, 1.5, 0, 0, 2, 0, 0, 0)^T$ and $\...
0
votes
1answer
68 views

Random number generator independence

I have a naïve question about random number generators. Say I want to generate a set of random variables from different exponential distributions,$$X_i \sim \mathcal {Exp}(\lambda_i)$$ Assume the$X_i$'...
1
vote
0answers
42 views

Pearson's Correlation between samples after PCA dimensionality reduction

This question is closely related to [Is it acceptable to reverse a sign of a principal component score?. I have a matrix A with Samples in row and Features in column. I apply PCA on A and get a ...
0
votes
0answers
24 views

Research on neural networks mimicking behavior

My question is regarding neural networks mimicking noise specifically. The idea. I have clinically clean image generated by a ray tracer and a neural network (type does not matter) that is mimicking ...
1
vote
0answers
30 views

How to determine statistical significance of physics dataset?

I am working on a physics paper, which looks at the dependence of resistivity on physical temperature. I obtained data showing a linear trend of the form $$\rho(T)=\rho_0[1+\alpha(T-T_0)]$$ Based on ...
0
votes
0answers
26 views

Setting seeds despite repeated training of CNNs?

I would like to compare the classification performance (like accuracy, precision, recall etc.) of different CNN architectures. I'm using Google Colab (GPU support), Tensorflow and Keras. Since it is ...
0
votes
1answer
44 views

How to compare the performance of different SVMs and CNNs?

I'm a beginner in machine learning und I have a problem to find the best way to compare the performance (accuracy) of different SVMs and CNNs (Jupyter Notebook)? The CNNs I train in Google Colab with ...
0
votes
1answer
564 views

How to compare CNN models with non-reproducible results?

I try to compare different CNN models. I use Keras and for training, I use a GPU, Google Colab with Tensorflow backend. Unfortunately I'm not able to create the same initial conditions for the CNNs (...
0
votes
0answers
59 views

Does a high negative concordance imply reproducible features

Does a high negative(-1) Lin's concordance correlation coefficient(Lins CCC) imply reproducible features ? I am calculating Lin's CCC of radiomic features between two different CT scan settings , ...
2
votes
1answer
118 views

Is most published research indeed false?

I have come across (Ioannidis, 2005) which explains several reasons (mainly statistics-related, that's why I post this question here) to justify the claim that most published research is indeed false. ...
2
votes
1answer
219 views

Reproducible numbers in Keras/TensorFlow

Every time I run a Keras/TensorFlow code gives different results. Can someone suggest how to get reproducible numbers?
9
votes
1answer
272 views

Why do people use PCA when it has so many issues?

(This is a soft question) Recently I'm learning Principal Component Analysis, and it appears to have a lot of issues: You have to transform the data to roughly the same scale before applying PCA, but ...
1
vote
0answers
40 views

Replicating a wrong analysis

A study measures the performance of patients and controls in two tasks. In task 1, a t-test reveals no significant differences between patients and controls. In task 2, a t-test reveals significant ...
4
votes
1answer
92 views

Why replication studies use two-tailed tests?

I have been searching replication studies and I found that all of them use two-tailed tests. For example: Open Science Collaboration. (2015). Estimating the reproducibility of psychological science....
3
votes
0answers
54 views

Has the reproducibility crisis affected confidence intervals as well?

The reproducibility crisis has given many pause over the value (?) of $p$-values to measure the relevance of statistical findings. Given the interpretation of a $p$-value and some knowledge of ...
0
votes
1answer
197 views

What is a good machine learning academic research workflow/tools? [closed]

I'm interested in ML research and want to get some insights into how its done in practice. In particular: What kind of data are you dealing with? How are you dealing with hyper-param optimization? ...
0
votes
0answers
26 views

Use Deep learning or RL to maximize long term hit in caching file?

Suppose I have F video files with different file size and with different popularity. I have a memory cache of fixed size. What I have to do is following: I want to cache n files (n < F) in ...
5
votes
2answers
8k views

Choosing the “Correct” Seed for Reproducible Research/Results

I have created this post to spark a discussion around an idea I had regarding the choice of the seed number used to replicate the results of a statistical model. Here is some background on how I came ...
2
votes
0answers
68 views

Put Together Results from GLM Gamma Models

I have a set of healthcare data in which I used GLM Gamma to model the healthcare spending. Then I am trying to put together the results in a style that is similar to this: http://onlinelibrary.wiley....
6
votes
1answer
314 views

“report the statistical results of the analysis without the covariate”

This is from Psychological Science 22(11), p.1363: If an analysis includes a covariate, authors must report the statistical results of the analysis without the covariate. Reporting covariate-...
0
votes
1answer
161 views

When to calculate Cronbach's alpha for a paper?

I am doing my undergraduate thesis, and I wonder if I need to do the Cronbach's alpha from my data set, or if I should use the Cronbach's alpha from the commercial questionnaire I will use for this ...
0
votes
1answer
120 views

I have all negatives in (some) of my folds in cross validation. Seeking best way to calculate AUC using R in this scenario?

Problem The title explains my problem. I cannot calculate AUC as things currently stand because some of my data sets have only negatives. I have three possible solutions and I need input from more ...
1
vote
0answers
95 views

Two samples t-test and ANOVA

B) When doing a two way ANOVA After calculating the sum of squares column(between), the sum of squares error(within), and the sum of squares block. 1. Are there any other sum of squares? 2. What is ...
0
votes
1answer
69 views

Simple questions about reproducibility

Suppose we made $m$ tests in 2 samples from the population. For simplicity, we compared $m$ traits (i.e. bmi, temperature, height, etc) for equality in means between 2 groups of people (i.e. Germans ...
1
vote
1answer
80 views

How to prove two methods are “similar”?

We are testing two imaging machines. One of the machines has been validated numerous times. However, we built the new machine and we need to prove that it is similar/comparable to the validated ...
0
votes
0answers
132 views

Calculate Cronbach's alpha for three observers

I have a dataset which contains 100 observations, and I asked three expert to describe each observation by one of the letters {A,B,C}. The following three tables shows a sample of the dataset with the ...
1
vote
1answer
45 views

Evidence for data mining (specification search) in published results [duplicate]

Is there an established way to assess the prevalence of data mining (as in specification search, not in the machine learning sense) in academic publications? I vaguely remember hearing something ...
6
votes
1answer
316 views

Statistical significance, repeatability, and sample size (50 shades of grey)

Reading about the problems with significance I wondered how one would approach an experiment in the right way, considering that in practice the sample size cannot be arbitrarily large. While some ...
94
votes
2answers
13k views

How much do we know about p-hacking “in the wild”?

The phrase p-hacking (also: "data dredging", "snooping" or "fishing") refers to various kinds of statistical malpractice in which results become artificially statistically significant. There are many ...
0
votes
1answer
141 views

How to calculate Cronbach's alpha

For my dissertation I am observing horse race videos to see if horses are right or left handed. In flat races where the horse comes out of the starting stalls I observe which leg (right or left) they ...
7
votes
1answer
645 views

Ethics of publishing covariate matrix

I am planning to submit a paper with a Structural Equation Model applied into a dataset built with measurements from schizophrenic patients. In the spirit of promoting an open research culture I've ...
3
votes
1answer
1k views

How can I define a multiple-record ID variable for a Cox model in R?

I'm reproducing and translating data from an old Stata file into R for future expansion and analysis. The original data uses a Cox proportional hazard model with ...
2
votes
1answer
74 views

Sample size for clinical research

I determined the sample size already (25), but I want to know if it has to be increased because of a minor change. Our study consists of two groups - one will be sedated and the other will not. For ...
0
votes
1answer
165 views

Data analysis: how to compare 2 variables from each subject across 3 groups of 200 subjects?

New here and not a statistician, am MD doing research. Need advice on best method for data analysis. I have 3 groups of 200 subjects each. We collected 2 data points (A,B) from each subject. Baseline ...
2
votes
0answers
94 views

How to organise an iterative manual rotation of n component pairs?

I am currently building a q.rotate() function for the qmethod R package for Q Methodology. As is desirable for Q, I'd like users to be able to iteratively rotate ...
15
votes
1answer
54k views

What if high validation accuracy but low test accuracy in research?

I have a specific question about validation in machine learning research. As we know, the machine learning regime asks researchers to train their models on the training data, choose from candidate ...
1
vote
1answer
106 views

What inputs, ideas or insight the community can offer on the subject “A simulation study of sample size for multilevel logistic regression.” [closed]

I have been assigned a topic on "A simulation study of sample size for multilevel logistic regression." I have searched the topic but found little reference on it. Could you please offer some ...