Even though I try to keep it as simple as possible, the pipelines for some of my data science projects get rather complex. At some point it becomes necessary to document this pipeline so that someone can return to the project, easily understand the various scripts and data-sources/outputs, and then update/modify it.
My question is: how do people document complex pipelines? For pipelines of medium complexity, I have take to drawing out the components of the pipeline with:
https://docs.google.com/drawings
Any suggestions for (or links to) best practices for documentation would be most appreciated.