Comparison of medians

Question

I`m currently an Intern and one of my tasks is to asses whether certain variables were optimal for a control system. In total there are four variables which can vary, and to avoid a complicated analysis I decided to, from the default settings (as a control), only vary one of these variables at a time and see how it impacted the stabilization time of the system. Each of these sample consists of 2500 tests and the time it takes to converge for each test. Just to clarify, my values are discrete as time is the number of cycles it takes to converge.

This approach seemed simple enough, however after preliminary analysis it became apparent that the distributions were not normal and that one of the parameters played heavily with the skew and shape of the PDF. Non-parametric tests seemed most-appropriate and I felt the median would be most representative of the time it took to settle, due to the heavy skew. The Mann-Whitney test seemed perfect, but the assumptions it required to be valid to compare the medians were too restrictive (My data is discrete and the distributions do not appear to just be shifted). The next test I thought of using was a permutation test to compare the medians of our two distributions, however I have no Idea how to conclude from the distribution of the medians. I though about just noting the percentage of the time that one settled faster than the other, but have no idea how to prove if that is statistically significant.

If additional information is required, please do not hesitate to ask. Thank you ahead of time!

Edit: Ok, minutes after having posted something clicked and now everything seems clear. My question now is rather does it make sense or is there a better approach?

Histograms

The first histogram is that of my control settings (epsilon = 0.1) and the other is when epsilon = 0.4. As epsilon grows the data's skew tends to become (from what I can see) more and more to the right. Due to my lack of reputation points I can only post two images, but the trends persists for all the values of epsilon I have tested thus far.

Answer 1

The description of the data would suggest that you have time-to-event data and that analysis with Cox's model could be employed to help resolve some of the distributional issues you describe. The issue here is not so much identifying a perfect distribution for your data as deciding what type of data you actually have. Based on the ability to perform 2500 repetitions of the experiment, please comment on whether your data are generated from a deterministic or stochastic process, which are less well-suited to statistical inference.

In order of preference, the negative binomial, Poisson or normal distributions are candidate distributions given your data. To use the normal distribution, consider a data transformation to reduce the skew. For each distribution, fit statistics are available to test the fit to the respective distribution.

Analysis of ranks can also be used, as you suggest in the original post. Note that ranks are being compared and not medians. For this sample size, non-parametric analysis would like perform as well as methods based on the normal distribution. The skew is not important for nonparametric analysis, as all values are ranked and analyzed.

Very generally, I would recommend a regression technique given the need to evaluate multiple independent variables. I would generally prefer a time-to-event analysis given the natural time to convergence you describe as the dependent variable. Cox's model or a parametric Weibull model (fit statistics available for both) may allow you to nicely evaluate your parameters using one test instead of multiple tests for values of each parameter, suggested in a previous version of the post.