5
$\begingroup$

How do I detrend or normalize multiple series of data so that I can inter-compare between the series?

Specifics below may not be appropriate for this forum. Please let me know and I can remove or re-phrase, but I think it might be helpful to fully understand the generic question above.

I have a data set that I would like help analyzing. I think this question belongs here and not in at https://gis.stackexchange.com/

Specifically, I have the following situation: Each series is collected from an airplane flying a path, and has a variable number of (value, lat,lon,time) tuples. I have several of these flightpaths, each at a different time, and flying different paths (sometimes crossing, sometimes not). Flights happened months apart, at different times of day, and due to natural phenomena, the data (thermal in this case) varies.

Part of the region flown over by multiple flights may or may not have an anomalous temperature signature. This is what I want to investigate. I am seeking an algorithm to detrend or normalize all of the flights so that I can increase my SNR and determine if there is a temperature anomaly over a sub-region.

Improve this question
$\endgroup$
2
  • $\begingroup$ It seems to me no valid comparison is possible among these series unless there is some persistent spatial phenomenon producing the "temperature signature." Otherwise they all measure different things. If there is a spatial pattern, then you really have a problem of spatial interpolation, not of time series normalization. $\endgroup$
    – whuber ♦
    Commented Sep 23, 2010 at 21:56
  • $\begingroup$ It is complex and messy real-world data. The "thermal anomaly" is (I hypothesize) variable in time but consistent in location. Overlaid on this is a different large-scale regional temperature gradient, seasonal and day/night variations, noise, instrument variability, etc. The flight paths come and go in different directions (so the mean temp of two flight paths should not be equal, even if season+time-of-day are removed, because of the large-scale regional gradient, etc.)... $\endgroup$
    – mankoff
    Commented Sep 25, 2010 at 7:14

1 Answer 1

Reset to default
6
$\begingroup$

Multi-level modelling where your data are grouped by flight as a random variable sounds like a good analysis method for this problem. In R the code might be

library(lme4) #load the package)
lmer(temp ~ region + (1|flight))

This is doable in a variety of statistics packages. If region is simply in region or outside of region then a logistic form should be used.

To directly address your question of normalizing you might like

temp - (mean_temp_for_flight - mean_temp)

This zeros the temperatures at the overall mean corrected for the individual flight mean. So, if a flight had a mean temp of 20 over the region, and the mean temp of the region is 18, and your sample is 22, then the normalized value would be

22 - (20 - 18) = 20

Essentially... by flight variability is eliminated.

Improve this answer
$\endgroup$
1
  • 1
    $\begingroup$ This is helpful, thank you. And addresses the simplified problem stated above. Unfortunately, as commented above, the real world is much more complex than what I initially described. I'm still working on it... $\endgroup$
    – mankoff
    Commented Sep 25, 2010 at 7:15

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.