I'm generally biased against overly prescriptive advice for modeling or how people should approach the craft of data, and as such, I would caution against thinking about this subject to the constraint of models/tests "you have to use". Like any technology choice, it comes down to trade offs. That said, my general inclination for your highlighted problems would be to convert the data to the appropriate statistically-digestible format (one-hot encode your categoricals), and probably use non-parametric bayesian approaches (i.e. MCMC) to determine the likelihood of "change" or "difference" in the underlying results. Although your data size is fairly limited, you could always go with traditional "p-hacking" statistics to assess significance as well, which is arguably a simpler approach.
Another approach to consider might be just throwing your cleaned data into a regression model (for absolute load time) or classification (pick an arbitrary threshold for 'slow' vs 'fast') and examine the feature importance characteristics of your model(s). This might be an easy way to spot-check whether certain features are have notable significance.
For #1 (impact of users on load time), to determine if users are a significant determinant, I'd start with a quick scatterplot of users vs load time (can do this for any hierarchical grouping of software version -- I'd probably start using all, then write a quick loop to iteratively go through each version to assess the existence of outliers or variation predicated along that dimension). My hypothesis would be that users would have fairly limited impact here, although might cluster (due to latency/bandwidth restrictions). If noteworthy, you could always extract a cluster assignment here and add them as input vectors for your modeling. Either way, should be able to "eye-ball" whether significance exists here and proceed accordingly. I'd recommend using something like bokeh or d3, where you can generate scatterplots with interactive tooltips and quickly indentify along which other dimensions user "groupings" might differ.
For #2 (impact of user | versions), I'd probably approach this in a similar way to #1, as the deconstructed problem is almost identical. A quick way to spot check could be to generate a correlation matrix on users vs. versions (with load time as the core metric), or just loop through each user and benchmark load time vs. version, looking for patterns. Most open-source ML packages should be able to tease this pattern out pretty easily. Another quick-and-easy way might be to control for users (by clustering into a few buckets [maintaining 90%+ variance or something], and then groupby those assignments), then look at timeseries of loadtime over version for each grouping. It may help to control for one of your variables here because, by the way you phrased your question, you're conflating them a bit.
For #3, I'd recommend grouping your data before endeavoring to plot this way (either by user cluster, or version, or any other features you extract) -- otherwise things can get out of hand quickly as your app and user-base grows. For a good example of a boxplot over time (or version #), I'd point you to this d3 example. I personally think boxplots can be misleading without the true distribution visual, and become a bit of an eye $*@& when too many are plotted adjacently. It sounds like your versioning is sequential (not overlapping A/B testing), so another approach could be to use the almighty linegraph (threw this together with arbitrary data):
Hope this helps, happy wrangling!