Adding noise to non-negative imputed data

Question

I have an hourly time series of wind speed data that spans 8 years. Wind speed must be non-negative but there are only ~80 out of ~70000 values that are exactly 0; the remainder are either positive or missing. There is a gap of a few months in the time series when the sensor was not functioning. I developed the following ad hoc method to impute the missing data (I understand this is not a great method for imputing values in such a long gap but I only need relatively plausible values to fill the gap). For every combination of month, day, and hour, I took the mean wind speed value at that hour across all years in a five-day window. For instance, if I wanted to impute the wind speed at 08:00 on 23 June, I would take the average wind speed at 08:00 on 21,22,23,24,25 June for all non-missing years in the dataset. Apart from whether this is a great method, it has a "smoothing" effect so that the variance of the imputed wind speeds is much lower than the observed wind speeds.

What I want to do is add noise to the imputed data so that the variance is equal to the variance of the time series overall. I thought I could take the following approach, based on the fact that the variance of the sum of two normally distributed random variables is the sum of their two variances:

• Find the standard deviation of the logarithm of the imputed values I: sd(log(I))
• Find the standard deviation of the logarithm of the observed values O: sd(log(O))
• Take the square root of the difference of the squared standard deviations: sd_diff = sqrt(sd(log(O))^2 - sd(log(I))^2)
• Generate Gaussian noise with standard deviation sd_diff: noise = rnorm(length(I), mean = 0, sd = sd_diff)
• Add this noise to the logarithm of the imputed values and back-transform: I_new = exp(log(I) + noise)

This approach, without the log transformation, worked for me to add noise to other imputed time series that were roughly normally distributed. However, when I use this approach with the log transformation, I end up with a higher standard deviation of the imputed values than of the observed values. I have the feeling that this relates to Jensen's inequality (log(sd(x)) < sd(log(x))). Is there any way that I can adjust the standard deviation of the noise so that I can achieve my goal of having the standard deviation of the imputed values match that of the observed values?

Answer 1

Note that the variation of the imputed unobserved values should be higher than of the really observed ones, because in addition to the variation the values would have even if you had measured them, you should also capture the uncertainty about what the values would have been, if they had been measured. The obvious way to get such a reasonable level of variation in the imputed values is to use multiple imputation for time series data (for which there are several available tools such as Amelia, which also has a R package - but in principle you could fit any Bayesian model and predict the missing days from the posterior predictive distribution).

The main trick would be to find good representations of the input data to capture the cyclic nature of the data (e.g. by creating categorical calendar month (or week) features, categorical "hour of the day" features, or perhaps cyclic spline representations of minute in the day, day of the year etc.) that let the imputation model realize that it should impute in part based on the same time periods in previous years and times on other days. Additionally, you might want to provide the imputation model with additional data that may not be as granular as the day you are interested in, but might help it (in part due to not being missing for the days for which you are imputing), e.g. ground-level wind speed predicted for the region by the national weather service, or perhaps national or regional wind power generation on the day or the week and things like that, which you can obtain.