2
$\begingroup$

I`m currently an Intern and one of my tasks is to asses whether certain variables were optimal for a control system. In total there are four variables which can vary, and to avoid a complicated analysis I decided to, from the default settings (as a control), only vary one of these variables at a time and see how it impacted the stabilization time of the system. Each of these sample consists of 2500 tests and the time it takes to converge for each test. Just to clarify, my values are discrete as time is the number of cycles it takes to converge.

This approach seemed simple enough, however after preliminary analysis it became apparent that the distributions were not normal and that one of the parameters played heavily with the skew and shape of the PDF. Non-parametric tests seemed most-appropriate and I felt the median would be most representative of the time it took to settle, due to the heavy skew. The Mann-Whitney test seemed perfect, but the assumptions it required to be valid to compare the medians were too restrictive (My data is discrete and the distributions do not appear to just be shifted). The next test I thought of using was a permutation test to compare the medians of our two distributions, however I have no Idea how to conclude from the distribution of the medians. I though about just noting the percentage of the time that one settled faster than the other, but have no idea how to prove if that is statistically significant.

If additional information is required, please do not hesitate to ask. Thank you ahead of time!

Edit: Ok, minutes after having posted something clicked and now everything seems clear. My question now is rather does it make sense or is there a better approach?

Histograms

The first histogram is that of my control settings (epsilon = 0.1) and the other is when epsilon = 0.4. As epsilon grows the data's skew tends to become (from what I can see) more and more to the right. Due to my lack of reputation points I can only post two images, but the trends persists for all the values of epsilon I have tested thus far.

Histogram of control settings Histogram of epsilon at 0.4

$\endgroup$
  • 2
    $\begingroup$ What do you intend by "non-parametric distribution" here? $\endgroup$ – Glen_b -Reinstate Monica Aug 4 '16 at 0:37
  • $\begingroup$ I meant that I didn't manage to fit it to a known distribution. I'll include some pictures, I think it will answer your question better than I can. $\endgroup$ – Felix Labelle Aug 4 '16 at 13:57
  • 1
    $\begingroup$ Even if you did fit a known distribution to your sample, it didn't actually come from that distribution; if the word "nonparametric" is simply to indicate we don't know what distribution(s) the data are from, it seems that it's not performing a useful role. We can almost never know that. $\endgroup$ – Glen_b -Reinstate Monica Aug 4 '16 at 16:21
  • $\begingroup$ Ok, how would you say that instead? My exposure to statistics has been rather limited thus far, so any help would be greatly appreciated! $\endgroup$ – Felix Labelle Aug 4 '16 at 17:53
  • 1
    $\begingroup$ Comparison of medians (if you really want a comparison of medians) or some similar terminology if you actually want to compare something else. $\endgroup$ – Glen_b -Reinstate Monica Aug 4 '16 at 22:36
0
$\begingroup$

The description of the data would suggest that you have time-to-event data and that analysis with Cox's model could be employed to help resolve some of the distributional issues you describe. The issue here is not so much identifying a perfect distribution for your data as deciding what type of data you actually have. Based on the ability to perform 2500 repetitions of the experiment, please comment on whether your data are generated from a deterministic or stochastic process, which are less well-suited to statistical inference.

In order of preference, the negative binomial, Poisson or normal distributions are candidate distributions given your data. To use the normal distribution, consider a data transformation to reduce the skew. For each distribution, fit statistics are available to test the fit to the respective distribution.

Analysis of ranks can also be used, as you suggest in the original post. Note that ranks are being compared and not medians. For this sample size, non-parametric analysis would like perform as well as methods based on the normal distribution. The skew is not important for nonparametric analysis, as all values are ranked and analyzed.

Very generally, I would recommend a regression technique given the need to evaluate multiple independent variables. I would generally prefer a time-to-event analysis given the natural time to convergence you describe as the dependent variable. Cox's model or a parametric Weibull model (fit statistics available for both) may allow you to nicely evaluate your parameters using one test instead of multiple tests for values of each parameter, suggested in a previous version of the post.

$\endgroup$
  • $\begingroup$ I hadn't thought of using a regression technique, I'll look into doing that! As far as using rank tests, I don't see the advantage of evaluating the ranks. From my understanding the only information that would give me is if my populations are different or not. As far as fiiting distributions, pretty much non of the samples have the same 'shape' if you will. Is it legit to use different statistical models for a hypothesis test? $\endgroup$ – Felix Labelle Aug 5 '16 at 16:27
  • $\begingroup$ I would focus on the regression method and not worry choosing models based soley on the shape of the data you obtained. In brief and as hinted at by @Glen_b, the test we use to evaluate the data obtained is often best determined by the data generating process, not the shape of the results generated from that process. You can only test a hypothesis using one model/test of the data if you are being strictly scientific in your approach. Reconciling two or more tests is not possible. $\endgroup$ – Todd D Aug 5 '16 at 16:50
  • $\begingroup$ After reading up a bit and thinking about it, I have somethings I'm unclear about using regression. I can't use each of my trials (one simulation) as a point in the regression due to the probabilistic nature of the system. The solution I came up with was using the medians of each series of tests with certain parameters. I was wondering if this seems valid? I'm also not sure that regression is exactly what I need. Although I'm interested in seeing the impact of my variables on my system, I really just need a way to compare the different results which I get after a series of trials. $\endgroup$ – Felix Labelle Aug 6 '16 at 15:13

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.