5
$\begingroup$

Background:

It's common in many aspects of science to want to determine if the trends shown in spatial data are significant. In my field of familiarity, climate science, authors often show the trends in temperature data, for example, and overlay stippling to show trend significance at 5%. For example see Rotstayn et al. (2013) (open access) Fig. 1:

Zonal-mean temperature trends with stippling denoting trends significant at 5%

Rotstayn et al. mention that

statistical significance of ensemble means is assessed using a two-sided t-test [Sect. 2]

For the example above, let's assume they have three data sets: model1 --> temperature climatology; model2 --> temperatures from an alternative model and model3 --> temperatures from another alternative model. (a) and (b) are then model2 - model1 and model3 - model1 respectively. In this case, the plot is latitude-pressure (height), but plots may be lat-lon, lon-pressure etc. Ultimately, you're working with a set of 2d matrices [x,y].

Question:

How do you obtain significance at each discrete data point rather than obtaining one t/p statistic for the comparison of the entire data set, such that you can plot of field of 5% significance.

I plot and process data in Python, so bonus points for discussion how this can be implemented in Python.

Also feel free to discuss the merits of t-tests and/or suggest alternative tests that may be better suited to data that is not normally distributed.

Note:

A similar question has been asked on Stackover flow here: Test for statistically significant difference between two arrays but I don't understand how the answers help me.

Improve this question
$\endgroup$
4
  • 2
    $\begingroup$ The answers on the SO thread do not help you because there you did not explain that the values in the arrays will exhibit strong spatial correlation. There is one answer there that correctly addresses the question you did ask: it is the one that points out the need to adjust for multiple testing. $\endgroup$
    – whuber ♦
    Commented Jun 19, 2014 at 14:35
  • $\begingroup$ @whuber Ok, thanks for the hint, but just to clarify, I didn't ask the SO question. Can you elucidate further on a work flow for the example above? The answer of user333700, to which I assume you are referring, suggests... Dividing the difference between gridpoints by the standard deviation, then gives t distributed random variables, that can be directly tested, i.e. the p-value can be calculated ... but tested how exactly? $\endgroup$
    – heds1
    Commented Jun 19, 2014 at 15:15
  • $\begingroup$ Sorry about confusing you with the other guy :-). That answer describes a Student t-test, that's all. But as I said, that won't directly help you. To get accurate answers you will need an accurate estimate of the spatial correlation function. $\endgroup$
    – whuber ♦
    Commented Jun 19, 2014 at 15:47
  • $\begingroup$ Some multiple p-value corrections are robust to positive correlation, they still maintain their size, but will or might be conservative, and will be less efficient or accurate than taking the spatial correlation into account. So it's still possible to estimate the trend for each point separately and correct the pvalues with, for example, a two stage FDR correction. $\endgroup$
    – Josef
    Commented Jun 25, 2014 at 14:37

1 Answer 1

Reset to default
1
$\begingroup$

You need to adjust for multiple testing (which you are doing when you plot a map with stippling: multiple hypothesis tests — 1 test per grid cell). I.e. you can't stipple grid cells whose p-values are lower than the typically used 0.05 significance level. You need to calculate a new reduced significance level ("p_fdr"; eqn. 3 in the paper provided) by controlling for the False Discovery Rate and only stipple grid cells whose p-values lie under that value.

The has been ignored in the atmospheric sciences for decades now. A recent publication focuses on just this exact issue: “The Stippling Shows Statistically Significant Grid Points: How Research Results are Routinely Overstated and Overinterpreted, and What to Do about It"

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.