11
$\begingroup$

I have a time series of measurements (heights-one dimensional series). In the observation period, the measurement process went down for some time points. So the resulting data is a vector with NaNs where there were gaps in the data. Using MATLAB, this is causing me a problem when computing the autocorrelation (autocorr) and applying neural networks (nnstart).

How should these Gaps/NaNs be dealt with? Should I just remove these from the vector? Or replace their entry with an interpolated value? (if so how in MATLAB)

$\endgroup$

4 Answers 4

4
$\begingroup$

I would not touch the data at all. Use this for autocorrelation with NaNs:

http://www.mathworks.com/matlabcentral/fileexchange/43840-autocorrelation-and-partial-autocorrelation-with-nans/content/nanautocorr.m

"not touch the data" means not to remove any data or time-step or replace with 0 or the mean, it would compromise the information about the specific-time-lag linear dependence. I would also avoid simulating the values in the gaps, if you are interested in the "SAMPLE" autocorrelation, anyway even the best simulation technique will not add any more information about the autocorrelation, being based on the data themselves. I partially recoded the matlab (link above) autocorrelation and partial autocorrelation functions to deal with NaNs: any data couples including NaNs is excluded from the computation. This is done for each lag. It worked for me. Any suggestion is well accepted.

$\endgroup$
2
  • $\begingroup$ Welcome @Fabio: Could you please give more explanation to what you mean with "not touch the data"? Do you mean not to remove anything? It would also help to introduce the content you linked to and explain why this helps the OP. $\endgroup$
    – Momo
    Oct 10, 2013 at 10:22
  • $\begingroup$ Hello Momo, thanks for the comment. "not touch the data" means not to remove any data or time-step or replace with 0 or the mean, it would compromise the information about the specific-time-lag linear dependence. I partially recoded the matlab (link above) autocorrelation and partial autocorrelation functions to deal with NaNs: any data couples including NaNs is excluded from the computation. This is done for each lag. It worked for me. Any suggestion is well accepted. $\endgroup$
    – Fabio
    Oct 21, 2013 at 17:47
3
$\begingroup$

There are some algorithms which are immune to missing values, so the preferred solution is to look for them (for instance R's acf for autocorrelation).

In general, the way to go is to either just discard data with missing observations (might be very painful) or just to impute their values -- mean of neighbors might be enough for smooth series and small gaps, but there are is of course of plethora of other more powerful methods, using splines, random/most frequent values, imputation from models, etc.

$\endgroup$
1
  • 2
    $\begingroup$ The acf with missing value is calculated in the normal way, but missing values are 'skipped' in the sums (that is, acf formula for a given lag looks like a sum divided by a sum, in each of those sums the missing values can be skipped). This is not the same as removing the missing values from the original data. The problem with matlab is it doesn't skip NaN, and including that in the calculation turns everything into NaN. $\endgroup$
    – Zero
    Apr 17, 2013 at 3:39
1
$\begingroup$

Use Intervention Detection to impute the missing vales exploiting the useful ARIMA structure and any local time trends and/or level shifts.

$\endgroup$
1
$\begingroup$

there are 2 problems here. the first is providing a meaningful numerical framework for your autocorrelation answer in matlab. for this to happen, you need to stretch and/or patch the time-series-portion of your data vectors...this 'data integrity' component of the problem is the most fundamental.

secondly, you need to decide how to handle the 'value' component of your vector...this depends to a large extent to the particular application as to what's best to assume, (e.g., small, missing time-stamps and the corresponding NaNs or Nulls could be safely interpolated from it's neighbors...in larger gaps, setting the value to zero is probably safer...or impute as recommended above--obviously for this to be meaningful, the gaps again must be comparatively small.).

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.