I designed a heuristic that solves a problem concerning network graphs. It was tested on thousands of different instances that have various different characteristics: Topology, template, number and position of users, capacities, ... It produced more than 300000 results that also depend on the random seed that was used.

In order to evaluate the results, I decided to use boxplots that I created with JFreeChart. I created different diagrams for the different topologies and with separate plots for every template. I felt that was a good way to visually summarize the results.

I was asked, why I didn't use confidence intervals instead to give an estimate. From what I know, these depend on a underlying distribution of a population parameter while boxplots don't. However, I summarize results of different seeds, numbers of users and capacities. All these influence the results. So I think I would not be possible to use confidence intervals unless I distinguished every single network characteristic.

Is that true? What are other advantages and disadvantages? And how could I argue, that I only use boxplots and not confidence intervals?

  • $\begingroup$ I see no "versus" here. Box plots show the entire distribution, summarized. You say you find them helpful. Confidence intervals arise when your concern is to estimate some parameter, say the mean of a variable, but quite possibly something else. In principle, the particular method you use to produce a confidence interval will depends on the underlying distribution, but with sample sizes $\sim 10^5$ that is not crucial. Nor does calculating a confidence interval for something depend on knowing everything in some sense: statistics would be impossible if so. $\endgroup$ – Nick Cox Feb 13 '15 at 10:44
  • $\begingroup$ You have both or either depending on what you are trying to do; it is just not clear what your aims are here statistically. You sketch what you've done but I can't extract more from your post than that you want to evaluate results. Descriptive statistics, including box plots, might easily be enough, or only part of the answer. $\endgroup$ – Nick Cox Feb 13 '15 at 10:47
  • $\begingroup$ I am evaluating the success rate, performance and quality of my heuristic. I also have the optimal results. I created boxplots for both to visually compare each of the three categories. I don't know if it makes sense to use confidence intervals here. Maybe to estimate the average runtime to calculate a result. I just don't really see the advantage of having a confidence interval for that compared to the boxplot. $\endgroup$ – CGFoX Feb 13 '15 at 10:53
  • 1
    $\begingroup$ Not trying to be flippant, but if you are uncertain about what you want statistically we can't tell you. With sample sizes such as you have, confidence intervals for the mean will be very narrow here, or so it would seem. I would be more concerned with the tails of the distribution and how bad results can be on occasion and the means won't tell you that directly (although clearly they will be affected by the extremes). (Detail: box plots don't usually show means, although many programs allow you to add means.) $\endgroup$ – Nick Cox Feb 13 '15 at 11:04
  • 1
    $\begingroup$ I'd much prefer quantile plots. $\endgroup$ – Nick Cox Feb 13 '15 at 12:48

Choosing box plots means that you print the 25th and 75th percentiles. Why not choose to print 2.5 and 97.5 percentiles? At n=300000 and unknown distribution that would be the most sensible definition of a confidence interval. You might even consider printing both in just one plot.

The purpose of the data evaluation is not perfectly clear and thus there is no better or worse to advise. If this is all about description, I personally feel that both descriptors contain too little of the available information. Have you considered violin plots? They might tell a lot more about the data's distribution than a boxplot or a confidence interval and take no more space than boxplots.

| cite | improve this answer | |
  • $\begingroup$ I agree in spirit, but for learners: the interval between any pair of quantiles for the data is best not thought of as a confidence interval for any parameter. You'll confuse others as well as yourself if you use this terminology, even if intended loosely and informally. By the way, there are several versions of box plots and similar displays that show much more information than the box and some whiskers. In climatology people used displays of all the data and quantile pairs other than the quartiles for decades before box plots were re-invented in mainstream statistics in the 1970s. $\endgroup$ – Nick Cox Feb 13 '15 at 13:05

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.