How can i compare Page Load Time of different websites I have 31 measurement values for load times of different web sites (100) under different configurations (A,B,C,etc.).
Say website A has a median Page Load Time (PLT) of 2000ms with configuration A (baseline) and a median PLT of 1700ms with Configuration B (optimization).
Now website B has a median PLT of 1000ms with configuration A and a median PLT of 1050ms with Configuration B.
I want to compare the effect of the optimization to the baseline variant in a sound manner. 
Obviously, absolute measurement values are different across websites. 
Originally, i thought about simply computing the speedup between the  median baseline case and the median optimization case.
However, as measurement values might be subject to noise, I am not sure whether I am allowed to do that, so what can i do?
The different configurations are part of a protocol optimization and are expected to yield benefits in terms of round-trip-time (i.e., 50-100ms).
We are running this experiments in a controlled laboratory setting, however due to computing times and variability of the browser and web server there is noise in between runs, which unfortunately is in the same range as the yields from our optimization.
The individual measurements are independent from another (ie., we always use a fresh browser without any intra-measurement effects). The effects that influence the load times are the design of the web pages itself, which may incurr processing.
 A: As you mentioned the effect of load time will be heavily dependant on the type of website. This will cause times of A and B to be small or large, but an issue overall because we now cannot compare them with the other websites. For this reason, you cannot use the mean or median time of the two features. Ideally, I would perform the additional processing and normalize my data against this metric. The metric could be the size of the website, amount of multimedia. Any factor that would heavily influence load time.
After normalizing the data, the only variable unaccounted for will be noise. Ideally, we would like to get a non-binary answer to your question "Is the optimized protocol better than the baseline case".
I simulated your problem by representing the baseline case as a normal distribution.
$$A \sim N(\mu_a,\sigma_a^2)$$
Let the improvement your protocol made be represented by $\mu_b$,$\sigma_b=0$
$$B = A+N(\mu_b,0)$$
Which makes,
$$B\sim N(\mu_a+\mu_b,\sigma_a^2)$$
B is however masked by an error term, you described to be comparable to the actual optimization.
$$error \sim N(0,\sigma_a^2)$$
$$B_{error} = B+error$$
$$B_{error}\sim N(\mu_a+\mu_b,2\sigma_a)$$
To measure how good your optimized protocol B is, you are trying to measure the probability,
$$Pr(B_{error}<A)$$
$$Pr(B_{error}-A<0)$$
This Distribution is given by,
$$B_{error}-A\sim N(\mu_b,3\sigma_a^2-cov(B_{error},A))$$
I simulated this on python:
import numpy as np
import seaborn as sns
from scipy.stats import norm

#Every row represents a website
A = np.random.normal(100,5,1000) #Assume this is the baseline
B_optimized = A+np.random.normal(-2,0,1000) #Assume this is the optimized
error = np.random.normal(0,5,1000)
B_optimized_error = B_optimized+error

sns.distplot(A, kde=False, rug=False);
sns.distplot(B_optimized, kde=False, rug=False);

A_mean = np.mean(A)
A_sd = np.std(A)

B_mean = np.mean(B_optimized_error)
B_sd = np.std(B_optimized_error)

import matplotlib.mlab as mlab
x_a = np.linspace(A_mean-3*A_sd,A_mean+3*A_sd,100)
x_b = np.linspace(B_mean-3*B_sd,B_mean+3*B_sd,100)

plt.plot(x_a,mlab.normpdf(x_a, A_mean, A_sd))
plt.plot(x_b,mlab.normpdf(x_b, B_mean, B_sd))

Z_sd = np.sqrt(np.var(A)+np.var(B_optimized_error)-2*np.cov(A,B_optimized_error)[0,1])
z = np.linspace((B_mean-A_mean)-10*Z_sd,(B_mean-A_mean)+10*Z_sd,100)
plt.plot(z,mlab.normpdf(z,B_mean-A_mean,B_sd+A_sd))

Z = B_optimized_error-A
d = norm(loc=np.mean(Z), scale=np.sqrt(np.var(Z)))
d.cdf(0)

You should get your answer by calculating the cdf at 0.
The results for $\mu_a=100sec$,$\mu_b=2$ $\sigma_a=5$ and a 10000 samples is as follows, Green is the improved protocol, Blue is the baseline,

This shows a 2 second improvement in the new protocol, with some noise added.

We find the probability of $B<A$ to be $68\%$
There are a lot of assumptions made in this model because it is simulated, I hope the precedure is clear. The most challenging step will be to normalize the data.
