# Comparing two datasets (of the same physical quantity) - what do I learn from this graph?

I'm having a hard time understanding what the authors of this paper (pdf) want to tell me with this graph (Fig. 2,3 (shown below) and 4, right):

[Caption: A comparison between the standard deviation of the differences (green line) and the standard deviation of all GOMOS (red line) and LIDAR (blue line) ozone profiles.]

Note: contrary to convention, the measured quantity is plotted on the x-axis, not y-axis.

The problem at hand is as following: Two instruments measure the same physical quantity (in this case vertical distribution of ozone in the atmosphere). Their corresponding (instrument characterized) standard deviations (averaged over if I understand correctly) are displayed in red and blue. The difference between the two datasets is computed and the standard deviation of the difference data set is plotted in green.

What does this graph mean? Is this a way to say that the two datasets are in agreement? If yes, on what basis?

What would other extremes mean?

• stdev of the difference is much larger than the measurement stdevs
I think: measurement stdevs are too optimistic.
• stdev of the difference is much smaller than the measurement stdevs
I think: measurement stdevs are too conservative.

Maybe an example would help me to understand this better. Thanks

-
If anyone could hint me towards some literature covering such comparisons, I'd greatly appreciate it. –  Sebastian Jun 9 '11 at 8:20

This appears to be an unconventional way to report correlation (or lack thereof). It focuses more on the variability of the measurements (across the earth at each fixed altitude) than on the correlation among them. As such the graphic may be of physical interest but it's an obscure way (at best) of comparing two measurement systems.

At each vertical position (an estimated altitude based on a pressure reading) the plots summarize between 3 and 18 pairs of data obtained over fixed stations on the earth's surface. The summaries consist of sample standard deviations normalized by the LIDAR readings (the reference measurement).

When comparing measurements, one is usually interested in assessing their correlations. We need to do a little math to relate this graphic to those correlations. Let $(X,Y)$ be a random variable representing the (LIDAR, GOMOS) readings. Let the variance of $X$ be $\sigma^2$, the variance of $Y$ be $\tau^2$, and their correlation equal $\rho$. Then

$$Var(X-Y) = Var(X) + Var(Y) - 2Covar(X,Y) = \sigma^2 + \tau^2 - 2 \rho \sigma \tau.$$

Consequently we can recover the correlation from the covariances:

$$\rho = \frac{1}{2\sigma \tau}(Var(X) + Var(Y) - Var(X-Y)).$$

Let the LIDAR mean be $m$. The plots depicts estimates of $\sigma/m$ (relative LIDAR SD): call this $s$; $\tau/m$ (relative GOMOS SD): call this $t$; and $\sqrt{Var(X-Y)}/m$ (relative SD of difference): call this $r$. Plug the estimates in to the preceding formula:

$$\rho = \frac{1}{2(s m)(t m)}((s m)^2 + (t m)^2 - (r m)^2;$$

$$\rho = \frac{s^2 + t^2 - r^2}{2 s t}.$$

These are, of course, estimates of $\rho$, subject to sampling uncertainty.

We can now qualitatively identify several portions of the plot:

• $s = t = r$, approximately, between 35 and 45 km. From the formula we estimate $\rho \sim 1/2$. This is modest correlation--not very good for two measurements of the same thing.

• One of $s$ and $t$ is small relative to the other and $r$ is comparable to the larger. This occurs from 20 to about 25 km and 45 to 50 km. The formula indicates $\rho \sim 0$. This is lack of correlation.

• $r$ is small and $s$ and $t$ are comparable (between 28 and 35 km). Now we estimate $\rho \sim 1$. This is what one hopes to see for two consistently comparable measurements.

In short, good correlation occurs when the green line lies substantially to the left of the red or blue lines and there is lack of correlation wherever the green line approximates (or exceeds) either or both of the red and blue lines. Overall, correlation is poor except between 27 and 34 km.

-
This is indeed a very good answer, @whuber. Would you recommend the usage of correlation is such cases? Would a plot altitude vs $\rho$ be clearer to everyone? –  Sebastian Jun 9 '11 at 8:06
@Sebastian That's on the right track. Going further, using about the same space as the triptychs that are shown, one would be able post a stack of scatterplots (of GOMOS vs. LIDAR), one for each kilometer or two. Or you could post a single scatterplot with the dots shaded by altitude. Or, on a single set of axes, you could plot a curve of GOMOS vs. LIDAR (parameterized by altitude) for each monitoring station. Any of these would be rich and revealing. –  whuber Jun 9 '11 at 13:47

Good questions. I scanned over the paper and have a couple of general thoughts...

First, with respect to

Note: contrary to convention, the measured quantity is plotted on the x-axis, not y-axis.

I like the unconventional orientation in this setting: with the Y-axis being altitude it lets me easily visualize that as I go higher into the atmosphere the new GOMOS instrument has less variable measurements of the same quantity as the traditional LIDAR method (and vice versa). This could be helpful in choosing instrumentation depending on where my particular data collection will take place, or may make me struggle with the idea of using two instruments (and their costs) to improve data quality.

Second, the left-most plots in Figures 2 and 3 visually show that the measurements are "very close", and the authors do call out the bias/variability differences with respect to altitude (which the plot you posted helps convey).

Finally, about the extremes, I think your "I think" statements imply that the two measurements are both valid, but it may be the case that at certain altitudes the bias and variability associated with a particular method means it should be abandoned in lieu of the other.

As for the statistical assessment of agreement, like you, I want to digest whuber's response.

-
The leftmost plots are indeed helpful, but as far as I can tell they compare average values (averaged over the monitoring stations), not individual values. Thus, their closeness would establish a lack of bias of one instrument relative to the other, but says little about their correlation. But the entire presentation is so...statistically unconventional (and terse) that I'm concerned this might be a misinterpretation. –  whuber Jun 8 '11 at 19:06
I work in lakes and lake sediments and putting depth on the y-axis (despite its being the independent variable) is actually the standard in papers in my field. My guess is that for atmospheric scientists this wouldn't be that unconventional either. As you suggest, it makes it so the figure is spatially accurate and therefore easier to read. –  KennyPeanuts Jun 10 '11 at 13:15