3
$\begingroup$

I am trying to correlate dendrochronological data with climate data. The first one is acquired directly from trees, the second one from various stations from around the world. According to the formula of Pearson correlation, two sets of values must be of the same size. But the climate data is not always complete - e.g. temperature might have not been collected on a given day 100 years ago.

What should I do in such a situation?

I had two ideas. Interpolate missing values or omit the incomplete pair. I don't want to do the first one as it artificially creates values which might not be true. But can I do the second one?

I am not a mathematician and I'm not sure whether it is a viable option. Also, if you had any sources to back your answers up, I'd appreciate it as well.

Improve this question
$\endgroup$
2
  • 6
    $\begingroup$ Interpolation might make sense otherwise, but can't help you gauge the strength of relationships between known data. Your best option is to use pairs for which both variables are known. (It's not obvious from what you say that Pearson correlation makes sense any way, but that's a different question.) $\endgroup$
    – Nick Cox
    Dec 28, 2015 at 11:30
  • 5
    $\begingroup$ I'd just like to emphasize and expand @NickCox final sentence: Correlation of time series data is often problematic or spurious. All sorts of time series can be correlated due to some other cause. $\endgroup$
    – Peter Flom
    Dec 28, 2015 at 12:23

1 Answer 1

Reset to default
3
$\begingroup$

Imputation (what you are calling interpolation) is widely used to handle missing data. You will obtain good estimates of Pearson correlation using (flexible) mean imputation. However, to estimate standard errors you will have to use multiple imputation. Omitting incomplete pairs is called a complete case analysis, and while inefficient, can work decently well. One must assume that the nature of the missingness doesn't depend on unmeasured values, like the outcome itself, for these approaches to be valid.

Improve this answer
$\endgroup$
3
  • 1
    $\begingroup$ Contrary to this, a contrast between interpolation and imputation is both clear and worth preserving. Interpolation is centuries old and historically was often focused on getting finer detail from mathematical tables (e.g. of logarithms). In a time series, interpolation uses only nearby values on the same variable, and can use e.g. linear, cubic, cubic spline methods (not at all a complete list). As I understand it, imputation usually (always?) uses relationships between variables to make predictions of what is missing, and typically (always?) does not use time or space ordering at all. $\endgroup$
    – Nick Cox
    Feb 26, 2018 at 17:31
  • $\begingroup$ @NickCox interestingly, air pollution modeling is an area where I've seen both principals applied. The goal is to predict individual level exposure to PM10 (or PM2.5) which is subject to seasonal variability and other exogenous time-varying or static trends. A flexible Kalman Filter is fitted capable of providing mean-level and individual level predictions over time. Most imputation techniques benefit from using simple panel data where the only assumed covariate structure is an exchangeable one, and even that is not turnkey as far as providing good predictions. Subtle and relevant point +1. $\endgroup$
    – AdamO
    Feb 26, 2018 at 17:39
  • $\begingroup$ Indeed. The principles are not exclusive. Oddly, I come across many fairly well trained statistically-minded people who have never heard of interpolation, beyond knowing from childhood, that you can join up the dots, whereas they are often very clued up on imputation (which has usually been oversold to them). Part of the explanation may lie in a decades-long division of teaching whereby interpolation is regarded as part of numerical analysis. (It was not always so, as witness older books on combination of observations.) Interpolation isn't magic either to repair broken data. $\endgroup$
    – Nick Cox
    Feb 26, 2018 at 17:45

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.