How to "normalize" standard deviations?

Question

I'm a computer science guy who's recently moved into Performance Engineering. As part of this job, I now find myself needing to analyze results of tests (duh). However, my lack of statistical knowledge is becoming limiting; I don't know what terms to use (and often misuse/conflate real statistics terms with colloquialisms), I don't know the veracity of my techniques for interpreting data, and I'm unsure how to "normalize" my data.

So for this question, I have a specific context, and I'd like to ask for the community's help in teaching me a little bit about what I think I need to do and how I should go about that. Remember that the terms I'm using may not correct in the statistical sense, so feel free to correct me when I misuse them.

So here's the context: I'm testing different environments with the exact same set of data and I want to analyze each test's individual results with the the other test permutations in order to see if specific parts of a test need further attention. So I have eight permutations of this test, where I perform the same exact set of tests with the same exact data for each permutation.

Here's a sample of what results would look like:

sessionId                           requestId                           requestNumber   result-01   result-02   result-03   result-04   result-05   result-06   result-07   result-08
e39bc31be83b4d67a3988798d3d1e485    f59210a1190a4860bd70e0726e12fe41    1               923         967         1061        1102        1285        755         1056        1043
1044aa10ea584cedb0f4b16fd4dcb875    d0f1546c4c7045efa531b7f8bad98f77    1               877         738         1033        1115        1221        692         718         1093
ac2323984be2414ab83bbc2a2ab75be9    fd46dbb899964fea9a3c29994a1d379d    1               874         1006        830         1090        841         722         755         1004
a211ad35b1bd4e8a97f3b7d33bb11f69    38b8a577f88d4a2da1696480d9509db1    1               678         1127        962         866         977         709         843         1027
eba78c9b753c4d50967413c5556ab805    b693824ec7a343d7ab93dce13a455463    1               830         1040        1306        1244        1038        726         918         1426
f2959209858140ffa78745165447c80c    9e8d256732f24e15b3dc67b5859e848e    1               779         875         1125        982         884         636         968         1776
e46bc6e847b64052832f786d873001bf    6c6f39a588354cc5bc8c411e2a02dc64    1               930         1106        848         1045        1279        709         1006        1753
e313caaf05774f2d8bd51334a24c5bbd    4404bd2d465043cdbbc12cbb527cc145    1               34          775         12          103         880         14          788         1445

Notice the last line of data has values for each test permutation that seem to vary rather wildly, at least compared to the results of the other data points.

Now, it's easy enough for me to calculate a standard deviation, but that in and of itself is rather meaningless, since it doesn't allow me to then filter and sort by it so that I know which particular tests I want to investigate further. For instance, one set of results may have a mean of 1,000,000 with a STDEV of 200, while another may have a mean of 400, again with a STDEV of 200. The former wouldn't be of concern, whereas the latter would.

In the example above, the last two data points would warrant further investigation since there are wildly different results.

What I think I want to do is "normalize" each line of data such that their standard deviations are on the same scale (e.g., 0..1 or 0..10). This would conceptually allow me to separate the data points that perform similarly across all eight test permutations from those that perform very differently across all, or a set of, the eight test permutations.

What are some ways I could accomplish this?

(If there's more context or information I can provide, please ask!)

Answer 1

I would use something simple. If you want to measure relative deviations within a session, you can calculate the coefficient of variation which basically measures what you described when you compared means to standard deviations (it is actually the ratio of standard deviation to the mean). The intuition behind cv is that it tends to be low for narrow distributions (such as a Gaussian with low standard deviation) and grows for wide distributions or distributions with long tail. Using your sample data, you get the following values for the rows:

cv[0] = 0.1392971502227546
cv[1] = 0.2059044516443989
cv[2] = 0.13708934947422577
cv[3] = 0.16106278913054062
cv[4] = 0.21328418997614207
cv[5] = 0.32100014994179166
cv[6] = 0.27541420818528267
cv[7] = 0.9984018893647104.

Clearly, the last session has a CV close to 1, meaning that the deviations in that case are comparable to the mean. In other words, the average is blurred by the large deviations.

Another approach could be to compare result-wise numbers in a session to the distribution you obtained for that same result (same column) in other sessions. In this case, you can use the z-score which is used in many applications related to Gaussian distribution, but you can use it to estimate deviations in each result column. The z-score effectively measures the deviation of your value from the population average in units of the standard deviation of that population. That is, if you already have a large deviation in a result, you can expect large differences. But it will tell you if a specific session has unexpectedly large deviation. That said, I first calculated the mean and standard deviation in each result column:

res_means = [740.625, 954.25, 897.125, 943.375, 1050.625, 620.375, 881.5, 1320.875]
res_std = [277.986, 136.124, 363.833, 333.640, 174.109, 231.403, 116.040, 303.365]

And then the z-scores for each session per result:

z_scores[0] = [0.656, 0.093, 0.45, 0.475, 1.346, 0.581, 1.503, -0.915]
z_scores[1] = [0.49, -1.588, 0.373, 0.514, 0.978, 0.309, -1.408, -0.751]
z_scores[2] = [0.479, 0.38, -0.184, 0.439, -1.203, 0.439, -1.09, -1.044]
z_scores[3] = [-0.225, 1.269, 0.178, -0.231, -0.422, 0.382, -0.331, -0.968]
z_scores[4] = [0.321, 0.629, 1.123, 0.901, -0.072, 0.456, 0.314, 0.346]
z_scores[5] = [0.138, -0.582, 0.626, 0.115, -0.957, 0.067, 0.745, 1.5]
z_scores[6] = [0.681, 1.114, -0.135, 0.304, 1.311, 0.382, 1.072, 1.424]
z_scores[7] = [-2.541, -1.316, -2.432, -2.518, -0.979, -2.62, -0.805, 0.409]

If you look at the values, the last session has many results with large absolute z-score values. As an illustration of this approach, you can count the number of results with absolute z-scores higher than a threshold, let's say 2. This will give you 0 for all sessions but the last, in which case you have 4 result columns with a large absolute z-score.