I'm a computer science guy who's recently moved into Performance Engineering. As part of this job, I now find myself needing to analyze results of tests (duh). However, my lack of statistical knowledge is becoming limiting; I don't know what terms to use (and often misuse/conflate real statistics terms with colloquialisms), I don't know the veracity of my techniques for interpreting data, and I'm unsure how to "normalize" my data.
So for this question, I have a specific context, and I'd like to ask for the community's help in teaching me a little bit about what I think I need to do and how I should go about that. Remember that the terms I'm using may not correct in the statistical sense, so feel free to correct me when I misuse them.
So here's the context: I'm testing different environments with the exact same set of data and I want to analyze each test's individual results with the the other test permutations in order to see if specific parts of a test need further attention. So I have eight permutations of this test, where I perform the same exact set of tests with the same exact data for each permutation.
Here's a sample of what results would look like:
sessionId requestId requestNumber result-01 result-02 result-03 result-04 result-05 result-06 result-07 result-08 e39bc31be83b4d67a3988798d3d1e485 f59210a1190a4860bd70e0726e12fe41 1 923 967 1061 1102 1285 755 1056 1043 1044aa10ea584cedb0f4b16fd4dcb875 d0f1546c4c7045efa531b7f8bad98f77 1 877 738 1033 1115 1221 692 718 1093 ac2323984be2414ab83bbc2a2ab75be9 fd46dbb899964fea9a3c29994a1d379d 1 874 1006 830 1090 841 722 755 1004 a211ad35b1bd4e8a97f3b7d33bb11f69 38b8a577f88d4a2da1696480d9509db1 1 678 1127 962 866 977 709 843 1027 eba78c9b753c4d50967413c5556ab805 b693824ec7a343d7ab93dce13a455463 1 830 1040 1306 1244 1038 726 918 1426 f2959209858140ffa78745165447c80c 9e8d256732f24e15b3dc67b5859e848e 1 779 875 1125 982 884 636 968 1776 e46bc6e847b64052832f786d873001bf 6c6f39a588354cc5bc8c411e2a02dc64 1 930 1106 848 1045 1279 709 1006 1753 e313caaf05774f2d8bd51334a24c5bbd 4404bd2d465043cdbbc12cbb527cc145 1 34 775 12 103 880 14 788 1445
Notice the last line of data has values for each test permutation that seem to vary rather wildly, at least compared to the results of the other data points.
Now, it's easy enough for me to calculate a standard deviation, but that in and of itself is rather meaningless, since it doesn't allow me to then filter and sort by it so that I know which particular tests I want to investigate further. For instance, one set of results may have a mean of 1,000,000 with a STDEV of 200, while another may have a mean of 400, again with a STDEV of 200. The former wouldn't be of concern, whereas the latter would.
In the example above, the last two data points would warrant further investigation since there are wildly different results.
What I think I want to do is "normalize" each line of data such that their standard deviations are on the same scale (e.g., 0..1 or 0..10). This would conceptually allow me to separate the data points that perform similarly across all eight test permutations from those that perform very differently across all, or a set of, the eight test permutations.
What are some ways I could accomplish this?
(If there's more context or information I can provide, please ask!)