I`m currently an Intern and one of my tasks is to asses whether certain variables were optimal for a control system. In total there are four variables which can vary, and to avoid a complicated analysis I decided to, from the default settings (as a control), only vary one of these variables at a time and see how it impacted the stabilization time of the system. Each of these sample consists of 2500 tests and the time it takes to converge for each test. Just to clarify, my values are discrete as time is the number of cycles it takes to converge.
This approach seemed simple enough, however after preliminary analysis it became apparent that the distributions were not normal and that one of the parameters played heavily with the skew and shape of the PDF. Non-parametric tests seemed most-appropriate and I felt the median would be most representative of the time it took to settle, due to the heavy skew. The Mann-Whitney test seemed perfect, but the assumptions it required to be valid to compare the medians were too restrictive (My data is discrete and the distributions do not appear to just be shifted). The next test I thought of using was a permutation test to compare the medians of our two distributions, however I have no Idea how to conclude from the distribution of the medians. I though about just noting the percentage of the time that one settled faster than the other, but have no idea how to prove if that is statistically significant.
If additional information is required, please do not hesitate to ask. Thank you ahead of time!
Edit: Ok, minutes after having posted something clicked and now everything seems clear. My question now is rather does it make sense or is there a better approach?
The first histogram is that of my control settings (epsilon = 0.1) and the other is when epsilon = 0.4. As epsilon grows the data's skew tends to become (from what I can see) more and more to the right. Due to my lack of reputation points I can only post two images, but the trends persists for all the values of epsilon I have tested thus far.