6
$\begingroup$

Even though I try to keep it as simple as possible, the pipelines for some of my data science projects get rather complex. At some point it becomes necessary to document this pipeline so that someone can return to the project, easily understand the various scripts and data-sources/outputs, and then update/modify it.

My question is: how do people document complex pipelines? For pipelines of medium complexity, I have take to drawing out the components of the pipeline with:

https://docs.google.com/drawings

Any suggestions for (or links to) best practices for documentation would be most appreciated.

Improve this question
$\endgroup$
4
  • 1
    $\begingroup$ I wonder if this question might be better suited for the Data Science Stack Exchange? $\endgroup$
    – Silverfish
    Jan 31, 2015 at 2:03
  • $\begingroup$ Didn't even know that there was such an exchange! thanks $\endgroup$
    – captain_ahab
    Jan 31, 2015 at 2:18
  • $\begingroup$ You might want to investigate drawing tools which can be programmed for larger scale pipelines. I tend to use Mathematica, but I'd guess there would be a suitable Python library if that fits with your skill set. $\endgroup$
    – image_doctor
    Jan 31, 2015 at 10:47
  • $\begingroup$ This is definitely an opinion-based question. But for what it is worth, I am finding that kedro provides a nice framework which has some self-documentation but also provides ready-to-use Sphinx boilerplate. $\endgroup$
    – Galen
    Aug 11 at 3:11

1 Answer 1

Reset to default
0
$\begingroup$

My 2p since this question has resurfaced to the top after few years with no answer...

In my opinion, it pays off to invest in a workflow manager. I'm very happy with snakemake and it's been a game-changer after having spent quite some time hacking together README files, bash scripts, and diagrams to keep track of complex pipelines.

Snakemake and similar tools are not easy to learn and depending on the complexity of your workflow, not easy to read either. However, they guarantee reproducibility and because you code input/output dependencies you don't need to rely on documenting what each step takes in input and output. Besides, snakemake has also options for reporting and printing the graph of job dependencies. I think that manually drawing the graph is a lot of work, error-prone, and difficult to keep in sync with the pipeline.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.