1
$\begingroup$

I'm evaluating different systems varying a certain parameter common to all of them. Let this parameter be x. At each x value I evaluate each system multiple times (eg: 10 times) and record the results. The issue is, with this number of evaluation (10 times) it's hard to get an idea about the underlying distribution of them. At some x values they look uniformly distributed but at some x values they look normally distributed. Apart from that in most cases there are outliers !!! And they also occur in one side ( like this - so it's skewed).

Now the issue is that I want to represent these results graphically. I had done similar plots in the past for normally distributed results as seen in the figure.x axis is the parameter <code>x</code>.

Different colors represent results for different systems. In each plot center line is the sample mean and the shaded area corresponds to $\pm2s$ (sample standard deviation). I would like to make a similar plot for these results as well. I have couple of ideas.

  1. Plot median as the center line and $Q1$ , $Q3$ as the shaded region. (optionally add outliers [ outside $3IQR$ from Q1 and Q3 ] like in box plots).
  2. Use a trimmed statistic (like 10% trimmed mean and standard deviation)
  3. Removing outliers and computing mean and SD as before (but I don't think it's a good idea).
  4. Fit a skewed distribution (like skew normal distribution). But not sure how I can represent it in the plot.

I'm planning to go with first option. Any suggestion is appreciated.

$\endgroup$
1
$\begingroup$

I would suggest bootstrapping your sample evaluations at each value of x to assign a measure of accuracy (in the form of a confidence interval) around each estimate.

Fundamentally, bootstrapping will allow you to estimate the distribution of the sample mean by repeatedly taking samples (with replacement) from your original sample. In your example, you only have 10 evaluations in your original sample at each value of x, so you would resample (perhaps 1000 iterations) from these 10 evaluations at each x to estimate a confidence interval around the mean.

Bootstrapping is particularly useful when the theoretical distribution of your parameter is complicated or unknown and when the sample size is insufficient for other methods. These criteria seem to fit your question which is why I would suggest this approach.

Below is a decent link for a simple example of how bootstrap sampling is executed to get a confidence interval:

http://statistics.about.com/od/Applications/a/Example-Of-Bootstrapping.htm

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.