Forgive me, I'm not very well seasoned in statistics and its visualization but I'm a well seasoned python developer and can use the basics of pandas, matplotlib, etc...
I have a dataset representing timings in milliseconds of a pipeline where each sample looks like (2 samples shown):
s0_begin 0 s0_foo 100 s1_foo_cut 189 d0_test_begin 1830 d0_test_end 3382 s0_end 3345 s0_begin 0 s0_foo 78 s1_foo_cut 164 d0_test_begin 1973 d0_test_end 6281 s0_end 6545
s0_begin is the starting timestamp (0), s0_foo is the first snapshot in milliseconds (it took 78ms to reach s0_foo in sample 2) and ends with s0_end.
I generate about 10K of these samples per day and I would like to visualize them to see where time is spent.
Since this data generation is automated and produces around 10K of these samples per day, it would also be helpful to visual how these change over time if possible: compare today to last month or yesterday, for monitoring and regression testing.
My main work is to try to optimize various parts of the pipeline to reduce duration so I'd like to be able to see these effects, i.e, if I reduced the d0_test duration by 20% then I should hopefully see this effect. What I've been doing so far is I've merely been producing individual histograms using .describe() in pandas for each slice: d0_test_begin - s1_foo_cut for example. This at least helps me to see if changes I make yield improvements.
I'm looking for advice on how to best visualize these samples intraday and also over time if there exists a reasonable way to do so. I will probably continue to dump histograms for each interval as well.