1
$\begingroup$

I have a multivariate time series (climate fluxes), and I have a number of model outputs that I want to asses. I also have number of metrics that I can use to assess the performance of each variable in the series (e.g. RMSE, mean bias, correlation, difference in variance, differences in skewness and kurtosis, differences in the upper and lower tails of the distributions, and so on - at least 10 metrics for each variable for model output).

So I end up with many, many scalar statistics for each model. Each of these numbers is a random variable, with it's own distribution, and those distributions are extremely diverse (e.g. RMSE can go to infinity, correlation's domain is $[-1, 1]$; bias is "better" as it goes to zero, correlation is "better" as it goes to 1).

I'm looking for a good method of visually representing the performance of each model using all of those metric results at once. I know that there's no perfect way to do this, because once you hit the Pareto region, you ultimately have to weigh the value of improved performance in one metric against degraded performance in another metric in a totally subjective way, but I'm basically just looking for ideas.

$\endgroup$

2 Answers 2

2
+25
$\begingroup$

A possible visualisation could be tree maps.

For each variable you can use correlation for the area and bias for the color. Variable that will be good with respect to both correlation and bias should be easy to spot.

In spite that, I'm not sure that visualisation in the most suitable direction here. Visualisation is powerful when you are dealing with few objects and few dimensions. Here you have many objects (your variables) and many dimensions (measures to apply on them).

You might find it more convenient to write all the measure into some database and just query all the variables that satisfy your constraints (correlation higher than x, variance lower than y, etc.)

$\endgroup$
1
  • 1
    $\begingroup$ Fair point about visualisation maybe not being suitable, I though as much, but thought I'd check first. Tree maps are an interesting idea, but I'm focussing more on trying to summarise across metrics than across variables, so I'd need more than two variables. $\endgroup$
    – naught101
    Aug 31, 2015 at 1:14
1
$\begingroup$

This does not seem to be a very multi-dimensional visualization problem: simple 3D bar plots should suffice (model/variable*metric). Although you do not mention the number of units you are trying to compare. As for different ranges, those can be standardized (or even inverted if a general 'good' direction is desired), although admittedly some information will be lost.

In addition, some of those metrics will be linearly related, so can be trimmed/simplified. Here's one example I find attractive, the one with transparency, although color can be reserved to be metric-specific, if this is the focus of the comparison.

$\endgroup$
2
  • $\begingroup$ Those plots only show one type of value over a grid. The problem is that the metric/variable combinations produce random variables that have vastly different distributions, so you won't get anything like the smoothness or comparability as you do with those plots. But yeah, so far barplots is about the the best option I can see (perhaps minmax or mean/variance normalised over models). $\endgroup$
    – naught101
    Sep 1, 2015 at 3:46
  • 1
    $\begingroup$ Right, that's what I meant by 'standardized'; furthermore you'd need to be very crafty in arranging/ranking them in such a way that is the most informative since transparency would only get you so far, and could also use color intensity to emphasize the best models. $\endgroup$
    – katya
    Sep 1, 2015 at 16:26

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.