0
$\begingroup$

I'm a computing science student and currently I'm working on a program that measures the execution time of certain pages on web-servers. In my program, it is possible to tag a version to your code, whenever you've changed anything in it.

Also, it stores which user loads the page on the web-server.

Thus, there are 3 dimensions to the data:

  • program-version (categorical)
  • user-id (categorical)
  • execution time (a sequence of numbers).

Questions:

  1. I want to test whether certain users have influence on the execution time of a web page. Which test do I have to use?

  2. I want to test whether some user perform better (i.e., have a different execution time) on certain versions. Which tests do I have to use?

  3. I want to create a graph for showing the execution time per user per version. Since the number of users can grow rapidly (currently around 100) and the number of versions also grows hard (currently around 20), a boxplot per user per version is too many. Is there another way of summarizing the data visually?

Thanks for helping me with my questions. I've been struggling with this for weeks now!

$\endgroup$
0
$\begingroup$

I'm generally biased against overly prescriptive advice for modeling or how people should approach the craft of data, and as such, I would caution against thinking about this subject to the constraint of models/tests "you have to use". Like any technology choice, it comes down to trade offs. That said, my general inclination for your highlighted problems would be to convert the data to the appropriate statistically-digestible format (one-hot encode your categoricals), and probably use non-parametric bayesian approaches (i.e. MCMC) to determine the likelihood of "change" or "difference" in the underlying results. Although your data size is fairly limited, you could always go with traditional "p-hacking" statistics to assess significance as well, which is arguably a simpler approach.

Another approach to consider might be just throwing your cleaned data into a regression model (for absolute load time) or classification (pick an arbitrary threshold for 'slow' vs 'fast') and examine the feature importance characteristics of your model(s). This might be an easy way to spot-check whether certain features are have notable significance.

For #1 (impact of users on load time), to determine if users are a significant determinant, I'd start with a quick scatterplot of users vs load time (can do this for any hierarchical grouping of software version -- I'd probably start using all, then write a quick loop to iteratively go through each version to assess the existence of outliers or variation predicated along that dimension). My hypothesis would be that users would have fairly limited impact here, although might cluster (due to latency/bandwidth restrictions). If noteworthy, you could always extract a cluster assignment here and add them as input vectors for your modeling. Either way, should be able to "eye-ball" whether significance exists here and proceed accordingly. I'd recommend using something like bokeh or d3, where you can generate scatterplots with interactive tooltips and quickly indentify along which other dimensions user "groupings" might differ.

For #2 (impact of user | versions), I'd probably approach this in a similar way to #1, as the deconstructed problem is almost identical. A quick way to spot check could be to generate a correlation matrix on users vs. versions (with load time as the core metric), or just loop through each user and benchmark load time vs. version, looking for patterns. Most open-source ML packages should be able to tease this pattern out pretty easily. Another quick-and-easy way might be to control for users (by clustering into a few buckets [maintaining 90%+ variance or something], and then groupby those assignments), then look at timeseries of loadtime over version for each grouping. It may help to control for one of your variables here because, by the way you phrased your question, you're conflating them a bit.

For #3, I'd recommend grouping your data before endeavoring to plot this way (either by user cluster, or version, or any other features you extract) -- otherwise things can get out of hand quickly as your app and user-base grows. For a good example of a boxplot over time (or version #), I'd point you to this d3 example. I personally think boxplots can be misleading without the true distribution visual, and become a bit of an eye $*@& when too many are plotted adjacently. It sounds like your versioning is sequential (not overlapping A/B testing), so another approach could be to use the almighty linegraph (threw this together with arbitrary data):

enter image description here

Hope this helps, happy wrangling!

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.