Forgive me, I'm not very well seasoned in statistics and its visualization but I'm a well seasoned python developer and can use the basics of pandas, matplotlib, etc...

I have a dataset representing timings in milliseconds of a pipeline where each sample looks like (2 samples shown):

s0_begin 0
s0_foo 100
s1_foo_cut 189
d0_test_begin 1830
d0_test_end 3382
s0_end 3345

s0_begin 0
s0_foo 78
s1_foo_cut 164
d0_test_begin 1973
d0_test_end 6281
s0_end 6545

s0_begin is the starting timestamp (0), s0_foo is the first snapshot in milliseconds (it took 78ms to reach s0_foo in sample 2) and ends with s0_end.

I generate about 10K of these samples per day and I would like to visualize them to see where time is spent.

Since this data generation is automated and produces around 10K of these samples per day, it would also be helpful to visual how these change over time if possible: compare today to last month or yesterday, for monitoring and regression testing.

My main work is to try to optimize various parts of the pipeline to reduce duration so I'd like to be able to see these effects, i.e, if I reduced the d0_test duration by 20% then I should hopefully see this effect. What I've been doing so far is I've merely been producing individual histograms using .describe() in pandas for each slice: d0_test_begin - s1_foo_cut for example. This at least helps me to see if changes I make yield improvements.

I'm looking for advice on how to best visualize these samples intraday and also over time if there exists a reasonable way to do so. I will probably continue to dump histograms for each interval as well.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.