6
$\begingroup$

I have got hourly temperature data from 2012 to 2016 as follows:

> head(htemp)
     HH_ID      TEMPERATURE   YY_ID    DD_ID       MM_ID
1 201201010000     8.98       2012    20120101     201201
2 201201010100     8.67       2012    20120101     201201
3 201201010200     8.69       2012    20120101     201201
4 201201010300     8.50       2012    20120101     201201
5 201201010400     8.30       2012    20120101     201201
6 201201010500     8.10       2012    20120101     201201

There are missing data in the data

  1. Missing data(NA) on individual hour (on 201209281400, 201209290000...)
  2. Missing data on consecutive hours(like no observation(NA) on 201210241700, 201210241800, 201210241900).
  3. No observation(NA) on a whole day(20130328).

I am wondering how to interpolate the missing data using adjacent data, i.e. linearly interpolation for individual missing HH data, and adopting the "typical" pattern from adjacent days for the whole day missing data (linearly interpolating each HH of the missing day using the temperature of corresponding HH in adjacent days).

Edit

I would like to perform the time series analysis on the temperature data, like decomposing (stl), modelling (auto.arima) and forecasting (forecast) it as well. It seems stl cannot handle missing data, so I think it might be necessary to impute the missing data first.

I have another data set containing electricity demand, where there is no missing data. I may also model the demand data using temperature data as covariate.

Update

I tried imp<-mice(htemp) on my data, but got an error:

iter imp variable
  1   1  TEMPERATURE
Error in solve.default(xtx + diag(pen)) : 
  system is computationally singular: reciprocal condition number = 5.03072e-28
$\endgroup$
5
  • $\begingroup$ I'd recommend using multiple imputation. Check out the MICE package. $\endgroup$ Jul 16, 2016 at 21:31
  • $\begingroup$ Hi @StatsStudent, I simply tried mice(), which does not work here. Would you mind giving some more hints? Thanks. $\endgroup$
    – Jeannie
    Jul 17, 2016 at 11:19
  • $\begingroup$ The error comes from the interrelated colums, e.g. MM directly follows from DD. For model based imputation, you would need to prepare the columns somewhat. $\endgroup$
    – Michael M
    Jul 17, 2016 at 11:43
  • $\begingroup$ You might also want to include the purpose of your overall analysis. Perhaps imputation is not the correct answer. $\endgroup$
    – Wayne
    Jul 17, 2016 at 13:14
  • $\begingroup$ @Wayne Thanks for your suggestion. I would like to perform time series on the temperature, and model another variable using temperature as covariate. It seems to me that imputing missing data at the very beginning will make the further analysis more convenient. $\endgroup$
    – Jeannie
    Jul 17, 2016 at 13:39

2 Answers 2

5
$\begingroup$

First thing, a lot of imputation packages do not work with whole rows missing. (because their algorithms work on correlations between the variables - if there is no other variable in a row, there is no way to estimate the missing values)

You need imputation packages that work on time features.

You could use for example package imputeTS to impute the temperature.

library(imputeTS)
x <- ts(htemp$TEMPERATURE, frequency = 12)
x.withoutNA <- na_kalman(x)

This would be one possible solution of getting imputed temperature values.

Here another one with the forecast package:

library(forecast)
x <- ts(htemp$TEMPERATURE, frequency = 12)
x.withoutNA <- na.interp(x)

These packages actually work, because they work on time correlations of one attribute instead of inter-attribute correlations.

$\endgroup$
1
$\begingroup$

You can also use package 'kssa'. It automatically help you to identify the best imputation method for your time series.

https://www.est.colpos.mx/web/packages/kssa/index.html

$\endgroup$
1
  • $\begingroup$ While this link may answer the question, it is better to include the essential parts of the answer here and provide the link for reference. Link-only answers can become invalid if the linked page changes. - From Review $\endgroup$ Jul 12, 2022 at 20:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.