Questions tagged [reproducible-research]

Research practice of making full experimental description, entire collected data, and all the data analysis scripts publicly available, so that the published results can be reproduced elsewhere.

1
vote
1answer
92 views

Is most published research indeed false?

I have come across (Ioannidis, 2005) which explains several reasons (mainly statistics-related, that's why I post this question here) to justify the claim that most published research is indeed false. ...
1
vote
1answer
31 views

Reproducible numbers in Keras/TensorFlow

Every time I run a Keras/TensorFlow code gives different results. Can someone suggest how to get reproducible numbers?
4
votes
0answers
74 views

Why do people use PCA when it has so many issues?

(This is a soft question) Recently I'm learning Principal Component Analysis, and it appears to have a lot of issues: You have to transform the data to roughly the same scale before applying PCA, but ...
0
votes
0answers
7 views

Comparing treatment outcomes

I am a statistics noob, my field is medicine, so I'm hoping someone can help me with this: I came across the following problem: "Researchers found that at the end of a 5-year study, 34% of the ...
1
vote
0answers
37 views

Replicating a wrong analysis

A study measures the performance of patients and controls in two tasks. In task 1, a t-test reveals no significant differences between patients and controls. In task 2, a t-test reveals significant ...
2
votes
1answer
42 views

Why replication studies use two-tailed tests?

I have been searching replication studies and I found that all of them use two-tailed tests. For example: Open Science Collaboration. (2015). Estimating the reproducibility of psychological science....
0
votes
0answers
22 views

Trouble replicating the experiments in “On-line Novelty Detection Using the Kalman Filter and Extreme Value Theory”

I'm trying to replicate the online novelty detection algorithm from "On-line Novelty Detection Using the Kalman Filter and Extreme Value Theory" by Hyoung-joo Lee and Stephen J. Roberts. In the first ...
0
votes
0answers
17 views

Calculation of adjusted CDS spread changes in event study

I am conducting a study about the impact of credit ratings on credit default swap spreads for European Sovereigns. In the related literature, most authors use adjusted spread changes to account for ...
0
votes
0answers
20 views

Statistics for Market research [duplicate]

I'm looking for some sources in order to learn about Market Research (a statistical approach with R/Python). You know, the kind of things you must know if you wanna work for Nielsen Kantar, etc.
0
votes
0answers
57 views

A question concerning the SHAP paper (A Unified Approach to Interpreting Model Predictions)

On page 6 of this paper, it is mentioned that the LIME model falls in the case of Equation 12 in that paper. My question is, in what sense? More specifically, Equation 12 approximates $f(h_x(z'))$ ...
3
votes
0answers
46 views

Has the reproducibility crisis affected confidence intervals as well?

The reproducibility crisis has given many pause over the value (?) of $p$-values to measure the relevance of statistical findings. Given the interpretation of a $p$-value and some knowledge of ...
0
votes
0answers
14 views

Replication, reproducibility and machine learning - reference request

Are there any papers where machine learning (NLP) was used to assess if papers would be reproducible? I remember something I think it was from a researcher in Harvard but I cant seem to find it.
0
votes
1answer
117 views

What is a good machine learning academic research workflow/tools? [closed]

I'm interested in ML research and want to get some insights into how its done in practice. In particular: What kind of data are you dealing with? How are you dealing with hyper-param optimization? ...
0
votes
0answers
25 views

Use Deep learning or RL to maximize long term hit in caching file?

Suppose I have F video files with different file size and with different popularity. I have a memory cache of fixed size. What I have to do is following: I want to cache n files (n < F) in ...
4
votes
2answers
3k views

Choosing the “Correct” Seed for Reproducible Research/Results

I have created this post to spark a discussion around an idea I had regarding the choice of the seed number used to replicate the results of a statistical model. Here is some background on how I came ...
2
votes
0answers
55 views

Put Together Results from GLM Gamma Models

I have a set of healthcare data in which I used GLM Gamma to model the healthcare spending. Then I am trying to put together the results in a style that is similar to this: http://onlinelibrary.wiley....
6
votes
1answer
251 views

“report the statistical results of the analysis without the covariate”

This is from Psychological Science 22(11), p.1363: If an analysis includes a covariate, authors must report the statistical results of the analysis without the covariate. Reporting covariate-...
0
votes
1answer
46 views

When to calculate Cronbach's alpha for a paper?

I am doing my undergraduate thesis, and I wonder if I need to do the Cronbach's alpha from my data set, or if I should use the Cronbach's alpha from the commercial questionnaire I will use for this ...
0
votes
1answer
72 views

I have all negatives in (some) of my folds in cross validation. Seeking best way to calculate AUC using R in this scenario?

Problem The title explains my problem. I cannot calculate AUC as things currently stand because some of my data sets have only negatives. I have three possible solutions and I need input from more ...
1
vote
0answers
85 views

Two samples t-test and ANOVA

B) When doing a two way ANOVA After calculating the sum of squares column(between), the sum of squares error(within), and the sum of squares block. 1. Are there any other sum of squares? 2. What is ...
0
votes
1answer
66 views

Simple questions about reproducibility

Suppose we made $m$ tests in 2 samples from the population. For simplicity, we compared $m$ traits (i.e. bmi, temperature, height, etc) for equality in means between 2 groups of people (i.e. Germans ...
1
vote
1answer
51 views

How to prove two methods are “similar”?

We are testing two imaging machines. One of the machines has been validated numerous times. However, we built the new machine and we need to prove that it is similar/comparable to the validated ...
0
votes
0answers
106 views

Calculate Cronbach's alpha for three observers

I have a dataset which contains 100 observations, and I asked three expert to describe each observation by one of the letters {A,B,C}. The following three tables shows a sample of the dataset with the ...
1
vote
1answer
43 views

Evidence for data mining (specification search) in published results [duplicate]

Is there an established way to assess the prevalence of data mining (as in specification search, not in the machine learning sense) in academic publications? I vaguely remember hearing something ...
6
votes
1answer
266 views

Statistical significance, repeatability, and sample size (50 shades of grey)

Reading about the problems with significance I wondered how one would approach an experiment in the right way, considering that in practice the sample size cannot be arbitrarily large. While some ...
94
votes
2answers
10k views

How much do we know about p-hacking “in the wild”?

The phrase p-hacking (also: "data dredging", "snooping" or "fishing") refers to various kinds of statistical malpractice in which results become artificially statistically significant. There are many ...
0
votes
1answer
111 views

How to calculate Cronbach's alpha

For my dissertation I am observing horse race videos to see if horses are right or left handed. In flat races where the horse comes out of the starting stalls I observe which leg (right or left) they ...
6
votes
1answer
623 views

Ethics of publishing covariate matrix

I am planning to submit a paper with a Structural Equation Model applied into a dataset built with measurements from schizophrenic patients. In the spirit of promoting an open research culture I've ...
2
votes
1answer
833 views

How can I define a multiple-record ID variable for a Cox model in R?

I'm reproducing and translating data from an old Stata file into R for future expansion and analysis. The original data uses a Cox proportional hazard model with ...
1
vote
1answer
62 views

Sample size for clinical research

I determined the sample size already (25), but I want to know if it has to be increased because of a minor change. Our study consists of two groups - one will be sedated and the other will not. For ...
0
votes
1answer
123 views

Data analysis: how to compare 2 variables from each subject across 3 groups of 200 subjects?

New here and not a statistician, am MD doing research. Need advice on best method for data analysis. I have 3 groups of 200 subjects each. We collected 2 data points (A,B) from each subject. Baseline ...
2
votes
0answers
91 views

How to organise an iterative manual rotation of n component pairs?

I am currently building a q.rotate() function for the qmethod R package for Q Methodology. As is desirable for Q, I'd like users to be able to iteratively rotate ...
13
votes
1answer
27k views

What if high validation accuracy but low test accuracy in research?

I have a specific question about validation in machine learning research. As we know, the machine learning regime asks researchers to train their models on the training data, choose from candidate ...
1
vote
1answer
103 views

What inputs, ideas or insight the community can offer on the subject “A simulation study of sample size for multilevel logistic regression.” [closed]

I have been assigned a topic on "A simulation study of sample size for multilevel logistic regression." I have searched the topic but found little reference on it. Could you please offer some ...
1
vote
1answer
82 views

Is golden section search considered a fitting (regression analysis)?

I am curious if the Golden section search is a form of fitting (or regression analysis in general)? It does find the extremum, but independent of any functional form (it works without modeling, if I ...
33
votes
5answers
5k views

Is p-value essentially useless and dangerous to use?

This article "The Odds, Continually Updated" from NY Times happened to catch my attention. To be short, it states that [Bayesian statistics] is proving especially useful in approaching complex ...
1
vote
1answer
688 views

Can't reproduce results using foreach and RSNNS

I'm trying to train multiple Multi-Layer-Perceptrons using foreach and mlp(...) from the RSNNS package. As I ran into problems reproducing the results generated by mlp() and foreach in a parallel ...
20
votes
1answer
3k views

Has the reported state-of-the-art performance of using paragraph vectors for sentiment analysis been replicated?

I was impressed by the results in the ICML 2014 paper "Distributed Representations of Sentences and Documents" by Le and Mikolov. The technique they describe, called "paragraph vectors", learns ...
2
votes
1answer
74 views

Publications in the social sciences or psychology based on freely available datasets?

There are a lot of datasets freely available across the web (see CV questions here, here, or here). Yet, I want the reverse: I am looking for a list of publications in the social sciences or ...
5
votes
1answer
4k views

Resources to write a statistical analysis plan

Following ICH-E9 guideline on statistical principles for clinical trials, A statistical analysis plan is a document that contains a more technical and detailed elaboration of the principal ...
1
vote
1answer
286 views

M statistic and software ProtoArray Prospector

I would like to reproduce the statistical result in this paper: http://www.ncbi.nlm.nih.gov/pubmed/24947363 They said "the M-statistic test in ProtoArray Prospector was applied to the normalized ...
3
votes
1answer
133 views

How should be statistics on scientific papers read?

Let's take this research published on Plos ONE Increasing carbohydrate intake was associated with increasing stroke risk (HR = 2.01, 95%CI = 1.04–3.86 highest vs. lowest quintile; p for trend 0....
4
votes
1answer
201 views

Standards on presenting graphs and charts in vector format?

In his up-voted answer to Software needed to scrape data from graph, Alexey writes (with respect to the need for data scraping off of raster graphs) "But nowadays the good practice is to publish ...
9
votes
4answers
1k views

Implications of current debate on statistical significance

In the past few years, various scholars have raised a detrimental problem of scientific hypothesis testing, dubbed "researcher degree of freedom," meaning that scientists have numerous choices to make ...
4
votes
4answers
2k views

Why does lambda.min value in glmnet tuning cross-validation change, when repeating test?

I'm using glmnet package to build a linear regression with $\alpha$ = 0.5, to find best $\lambda$. xMatrix [44x15000] ($p>>...
5
votes
1answer
102 views

Licensing of analytical code and data - which license to use, and what issues?

Not sure if this belongs to CrossValidated, Computational Science, or something else entirely. I've got some code and data that's going into a publication, but I'd like to make said code and data ...
12
votes
3answers
234 views

Hosting options for publicly available data

So you've decided to support the idea of reproducible research and want to make your data available online for people to see and use. The question is, where do you host it? My first inclination is of ...
26
votes
2answers
15k views

What are some standard practices for creating synthetic data sets?

As context: When working with a very large data set, I am sometimes asked if we can create a synthetic data set where we "know" the relationship between predictors and the response variable, or ...
6
votes
5answers
847 views

Comparing reproducible research strategies: brew or Sweave vs. R2HTML

This is a bit more "think about it" question - but I see it as an important one to ask. I have been struggling for the past few days with having a more reproducible-research-like workflow. I am ...
50
votes
3answers
3k views

How are we defining 'reproducible research'?

This has come up in a few questions now, and I've been wondering about something. Has the field as a whole moved toward "reproducibility" focusing on the availability of the original data, and the ...