Looking for input on whether I'm using the right tool for the job. In particular, I'm currently using bootstrapping but am beginning to suspect I should be using something else.
Background on problem domain
Whenever your web browser requests an item (whether it be a web page, video, image, etc.), the content provider (YouTube, Reddit, etc.) may be able to send the response via different Internet routes. I'm currently running an experiment where traffic is spread randomly across routes. My goal is to identify cases where there is a significant difference between routes for the same end users.
When I say significant, I'm referring to a difference in a metric that a user could perceive: for instance, a 20 ms difference in latency (the amount of time for a message to flow from a client to the server).
One of the largest problems I face is the amount of "noise" in the data. For instance, a microwave being on in someone's house can cause their Wi-Fi connection to become degraded, resulting in increased latency measurements. I've found that this makes metrics like 'average' useless. It also reinforces the fact that I need to have an understanding of uncertainty in the measurements -- I should not just be directly comparing the P50 or average latency of one route to another.
Since the data is not normally distributed, my plan was to use bootstrapping to generate confidence intervals for a given metric for each route. When comparing the performance of two routes, I would then calculate the difference between their confidence intervals and see if this was greater than or equal to the threshold I set for the metric.
However, it's not immediately clear this is the right thing to be doing. In particular, if I have two routes and their confidence intervals are just barely overlapping, it seems like it may still make sense to assume that one of the paths is better than the other (perhaps this is effectively where you adjust the confidence interval size).
An alternate seems to be bootstrapping the difference between the metrics (so bootstrapping the difference between the medians, etc.). This is a little bit closer, as it is giving me an interval of possible ranges. One concern I have with this is that I recall reading that both sample sizes must be the same for this to work (is this true?)
In general, one of the problems I've had with bootstrapping is that I cannot figure out how to use the result for making a decision. Ideally, I'd like to be able to say "I am X% certain that route A is better than route B by at least Y ms", and if X > some threshold, take an action. But with bootstrapping, I'm getting confidence intervals -- it's not clear how I can translate those into the same information.
In comparison, Monte Carlo simulations make more sense to me here -- I simulate each route's performance and record whether A was better than B by at least Y ms. After completing the simulation, I check if A was better than B for X% of the time -- if so, I take action. However, to my knowledge, I cannot use resampled data to execute a Monte Carlo simulation.
It would be great to get some insight into different options and what I should look into further. In particular, if bootstrapping isn't the right tool I'm happy to switch away, but it's clear to me that I need some way to understand the uncertainty in these measurements. Thanks!