0
$\begingroup$

I have two data sets, both ranging from 1996-2016. However, the y-axis values are on completely different scales. The first is for mean NDVI values where 0 is centered on the mean (.1865) and the ranges are the differences between the mean and the values for that year that range from -0.03 to 0.03. The second dataset is the Palmer Drought Severity Index with the same date range, but range from -7 and 6.

I want to know the best way to find correlation between these two datasets, and would prefer to be able to do it in python. Here is an image of the two plots, both have the same x-axis which is the year range from 1996-2016 Here are the two data set plots, bot have the same x-axis which is years ranging from 1996-2016

$\endgroup$
  • $\begingroup$ Do you want to expand on your statistical question? Are you asking whether ti is appropriate to correlate values from two time series? Asking for code is off-topic here. $\endgroup$ – mdewey Nov 30 '16 at 16:57
  • $\begingroup$ I was under the impression it was appropriate, but I would like to know how to do it. Would Pearson's correlation work, or is there something better? $\endgroup$ – Vanludvig Nov 30 '16 at 16:59
  • $\begingroup$ interpreting correlations between two time series can be tricky: if the two series share similar autocorrelations, then the correlation between them can be inflated. There's a whole literature on prewhitening in time-series analysis, e.g. agupubs.onlinelibrary.wiley.com/doi/full/10.1029/2003WR002073 $\endgroup$ – Ben Bolker Mar 28 '18 at 16:52
1
$\begingroup$

The different ranges of your data is no problem, since e.g. scaling (both or just one of them) to mean zero and unit variance does not change the correlation between them. However, if you want to correlate the two data vectors you have, the need to have the same length, i.e. if your PMDI vector has more data points than your other vector, then you need to find a way (e.g. taking the mean over some period) to summarise your PMDI vector in less data points. Calculating correlation in Python: See e.g. https://stackoverflow.com/questions/19428029/how-to-get-correlation-of-two-vectors-in-python

$\endgroup$
0
$\begingroup$

As Schaftablette mentioned, for (Pierson or any other) correlation, or even for correlation plot, keeping the values within the same range is not necessary. But rescaling both vectors to the same range (say [0,1]) would give you a cleaner plot.

What is crucial in your case ,however, is the different size of your datasets. For correlation analysis, you need to have equal size of data in both dimensions. As your plots show, they are not equal. So, you need to resample your data to make them equal. You can either go with under-sampling of the larger dataset, or, over-sampling of the smaller one. I would suggest a k-fold random under-sampling. Suppose your smaller dataset has m data points. Then you can randomly sample from your larger dataset, and pick m of them and find the correlation. If you repeat this procedure k times and get the average, your final result would be fairly reliable. Of course, this is recommended only if you do not have access to more samples of your smaller dataset.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.