How to deal with gaps/NaNs in time series data when using Matlab for autocorrelation and neural networks?

Question

I have a time series of measurements (heights-one dimensional series). In the observation period, the measurement process went down for some time points. So the resulting data is a vector with NaNs where there were gaps in the data. Using MATLAB, this is causing me a problem when computing the autocorrelation (autocorr) and applying neural networks (nnstart).

How should these Gaps/NaNs be dealt with? Should I just remove these from the vector? Or replace their entry with an interpolated value? (if so how in MATLAB)

Answer 1

I would not touch the data at all. Use this for autocorrelation with NaNs:

http://www.mathworks.com/matlabcentral/fileexchange/43840-autocorrelation-and-partial-autocorrelation-with-nans/content/nanautocorr.m

"not touch the data" means not to remove any data or time-step or replace with 0 or the mean, it would compromise the information about the specific-time-lag linear dependence. I would also avoid simulating the values in the gaps, if you are interested in the "SAMPLE" autocorrelation, anyway even the best simulation technique will not add any more information about the autocorrelation, being based on the data themselves. I partially recoded the matlab (link above) autocorrelation and partial autocorrelation functions to deal with NaNs: any data couples including NaNs is excluded from the computation. This is done for each lag. It worked for me. Any suggestion is well accepted.

Answer 2

There are some algorithms which are immune to missing values, so the preferred solution is to look for them (for instance R's acf for autocorrelation).

In general, the way to go is to either just discard data with missing observations (might be very painful) or just to impute their values -- mean of neighbors might be enough for smooth series and small gaps, but there are is of course of plethora of other more powerful methods, using splines, random/most frequent values, imputation from models, etc.

Answer 3

Use Intervention Detection to impute the missing vales exploiting the useful ARIMA structure and any local time trends and/or level shifts.

Answer 4

there are 2 problems here. the first is providing a meaningful numerical framework for your autocorrelation answer in matlab. for this to happen, you need to stretch and/or patch the time-series-portion of your data vectors...this 'data integrity' component of the problem is the most fundamental.

secondly, you need to decide how to handle the 'value' component of your vector...this depends to a large extent to the particular application as to what's best to assume, (e.g., small, missing time-stamps and the corresponding NaNs or Nulls could be safely interpolated from it's neighbors...in larger gaps, setting the value to zero is probably safer...or impute as recommended above--obviously for this to be meaningful, the gaps again must be comparatively small.).