Without going into the details of a statistical simulation that I am working on, I would like to ask for advice for the following problem.

I am simulating the mean sqaured error (MSE) of a set estimators under many conditions, most of which can be chosen on continuous scales, which are constrained to the $[0,1]$ interval (e.g. correlations between variables in the simulation). For different discrete values (say $[0,0.1,0.2,...,1]$ for each variable in the simulation) and their fully crossed combinations I simulate and obtain the MSE of all estimators.

The results of such simulations are easy to interpret and illustrate (e.g. visualize) as long as we are dealing with a small number of conditions (say, one or two). An obvious approach is to plot contuors of MSE for the levels of conditions. For higher numbers of conditions, we have an essential problem which is akin to multivariate data analysis with one predicted variable (MSE) and multiple predictors (conditions) in an experimental setting.

I am in the situation that I want to compare the results (MSE) under a moderately large number of conditions (say, four or five) across a number of different estimators (say, ten). What are useful ways to describe such results, e.g. in academic papers? Clearly it is impossible to report MSE for all combinations of conditions and estimators, nor are visual plots straight forward. How can I discover patterns in my simulation results?

Two thoughts of mine on this problem are:

  1. Use MSE as a predicted variable and the conditions as predictors with interactions in a regression model. Interpret coefficients as sensitivity of condition to change the MSE.
  2. Estimate the "variance of MSE" between conditions and across all conditions, plus mean MSE across all conditions. Larger variance (or standard deviation) of MSE across conditions indicates that under some conditions MSE is 'high'.

I'd be also interested in visual approaches or papers discussing presentation of simulation studies.


One way to do up to 4 dimensions, is to have a 2D array of graphs. It can stretch readability a bit, but if you have a full page and each graph is relatively simple, that might work. Seaborn in Python does this nicely, for example:

http://stanford.edu/~mwaskom/software/seaborn/examples/faceted_histogram.html http://stanford.edu/~mwaskom/software/seaborn/examples/many_facets.html

Another option for dimensionality reduction would be to do principal components analysis on the MSE, and then graph the contour map against those the first two PCs.

I think the answer to your question in particular is highly dependent on the distribution of the MSE. If the distribution has a single global minimum and few local minima, I'd graph the MSE through that point against the two most informative axes, and then have separate graphs to explain the effect on MSE of each of the other variables. If there's some particularly interesting relationship between two of the predictors, that'd be worth a graph as well.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.