Stack Exchange Network

Stack Exchange network consists of 175 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [project-management]

Organizing computational work on *statistical* projects; use for questions about data storage, data sharing, code repositories, etc. Note that questions about programming or unrelated to statistics are off-topic.

0
votes
1answer
517 views

best practice: preprocessing holdout set at same time as train set, or no?

I am a very new ML programmer, and I have come across a dilemma regarding best practices. Things will "work" either way for me, but I want to know what the best practice is. I am performing text ...
3
votes
1answer
365 views

Why R markdown / python Jupyter are widely used in statistical projects?

I personally hate R markdown file / python notebook, since they mix code and results all together (expose unnecessary details / show pre-mature progress in exploration stage). But I noticed there are ...
1
vote
0answers
72 views

Machine Learning Organization/Object-Oriented ML

How do you guys organize your ML/data science code and rapidly explore new options? I took an OOP course in college, but I've been at a loss as to how to structure my code. For example: I have ...
8
votes
5answers
178 views

Preserving comments on graphs for exploratory data analysis

In performing exploratory data analysis, I will often print out the graphs and write out comments/annotations etc. Do people have suggestions for a better electronic methodology? I am especially ...
2
votes
1answer
952 views

Is there an formula to translate a Beta distribution back to a Pert distribution? Or is simulation the only way?

A popular method to estimate in project management is the 3 point estimation plus the PERT formula If I use it for example to estimate how many stories my team is going to be able to complete Sprint ...
1
vote
0answers
41 views

Importance of workflow visualization in a statistical software package?

From a blog post ... The important thing is why they’re missing. Gartner excluded them this year, “…due to not satisfying the [new] visual composition framework [VCF] inclusion criteria.” VCF is ...
4
votes
1answer
164 views

How can I use bayesian reasoning to explain a Scrum team if they initial estimates were right or wrong and if the project is currently delayed or not?

Imagine a software development team estimates they are going to be able to complete 80 user stories in 5 sprints using Scrum: Sprint 1: 16 stories Sprint 2: 16 stories Sprint 3: 16 stories Sprint ...
32
votes
7answers
10k views

Why is a comma a bad record separator/delimiter in CSV files?

I was reading this article and I'm curious for the proper answer to this question. The only thing that comes to my mind it's perhaps that in some countries the decimal separator is a comma, and it ...
1
vote
1answer
58 views

(file) dependency tracking in Stata

As much of empirical research (or data analysis) has become a major software development exercise, practitioners use more and more such tools. So far I am aware of researchers using project management,...
21
votes
5answers
3k views

How to keep exploratory analyses of large datasets in check?

When I start an exploratory analysis on a large data set (many samples, many variables), I often find myself with hundreds of derived variables, and tonnes of different plots, and no real way to keep ...
5
votes
1answer
427 views

Efficient data flow for R?

There are already some good answers on project management (eg. How to efficiently manage a statistical analysis project?). These are great, and make life a lot easier. In particular, the workflow that ...
2
votes
1answer
211 views

Dealing with project uncertainties: is the sum of the most-likely estimates equal to the sum of the expected times?

I found this document on the internet: Dealing with Project Uncertainties In this article I read: In order for uncertainties to be included in the project estimates it is necessary to take ...
2
votes
2answers
5k views

Calculating probability of project tasks, and most likely ending date?

Say I have a project (you know the typical one you can represent using a Gantt Chart ) If I where to assign a particula probability to each of the tasks in the Gantt... say for example: Task1 (2 ...
4
votes
1answer
160 views

Best practice when merging waves from non-panel survey

This is more of a data management question. When merging several waves of a survey where the same questions were asked in multiple years -- but each time of a new sample -- should the data from a ...
10
votes
3answers
305 views

Improving variable names in a dataset

Good variable names are: a) short / easy to type, b) easy to remember, c) understandable / communicative. Am I forgetting anything? Consistency is something to look for. The way I would put ...
15
votes
3answers
4k views

What is a practically good data analysis process?

Maybe this is a dumb question as my experience has been more on statistical computing algorithms than on data analysis. But I would like to know, or have references on, analysis process most of ...
4
votes
0answers
170 views

What is the Johnson family of distributions good for? What are their main features and applications?

I saw the Johnson family of distributions in context of reliability and demand modeling for supply chains, but I am not sure if they are bringing real benefit to pay for their relatively more complex ...
8
votes
5answers
745 views

What is a good general purpose plain text data format like that used for Bibtex? [closed]

Context I'm writing a few multiple choice practice questions and I'd like to store them in a simple plain text data format. I've previously used tab delimited, but that makes editing in a text editor ...
6
votes
5answers
847 views

Comparing reproducible research strategies: brew or Sweave vs. R2HTML

This is a bit more "think about it" question - but I see it as an important one to ask. I have been struggling for the past few days with having a more reproducible-research-like workflow. I am ...
7
votes
3answers
3k views

Workflow for using R with LyX for a statistical analysis? [closed]

For some time, I wanted to stop copy-pasting my R results into word, but climbing the LaTex mountain seemed to much to be worth it. Recently, I came to discover LyX, as a laymen's solution to people ...
5
votes
2answers
317 views

Project management for remote collaboration in prediction

Are there any tools for remote collaboration in prediction or machine learning settings? I am looking for a computing environment that includes appropriate source control, keeps track of how ...
23
votes
4answers
17k views

What are efficient ways to organize R code and output? [closed]

I am looking for input on how others organize their R code and output. My current practice is to write code in blocks in a text file as such: ...
8
votes
1answer
987 views

Statistical project directory structure with multiple languages (e.g., R and Splus)?

Building on the post How to efficiently manage a statistical analysis project and the ProjectTemplate package in R... Q: How do you build your statistical project ...
17
votes
5answers
662 views

Simple, reliable, open, and interoperable plain text format for storing data

In a previous question I asked about tools for editing CSV files. Gavin linked to a comment on R Help by Duncan Murdoch suggesting that Data Interchange Format is a more reliable way to store data ...
18
votes
10answers
12k views

Strategy for editing comma separated value (CSV) files

When I work on data analysis projects I often store data in comma or tab-delimited (CSV, TSV) data files. While data often belongs in a dedicated database management system. For many of my ...
7
votes
2answers
646 views

Repositories and data analysis projects

Context: I've recently adopted version control as part of my data analysis work (finally I may hear you saying: see my earlier question on SO). This prompted me to think more about repositories and ...
31
votes
6answers
1k views

How to increase longer term reproducibility of research (particularly using R and Sweave)

Context: In response to an earlier question about reproducible research Jake wrote One problem we discovered when creating our JASA archive was that versions and defaults of CRAN packages ...
89
votes
7answers
21k views

How to efficiently manage a statistical analysis project?

We often hear of project management and design patterns in computer science, but less frequently in statistical analysis. However, it seems that a decisive step toward designing an effective and ...