I have an hourly time series of wind speed data that spans 8 years. Wind speed must be non-negative but there are only ~80 out of ~70000 values that are exactly 0; the remainder are either positive or missing. There is a gap of a few months in the time series when the sensor was not functioning. I developed the following ad hoc method to impute the missing data (I understand this is not a great method for imputing values in such a long gap but I only need relatively plausible values to fill the gap). For every combination of month, day, and hour, I took the mean wind speed value at that hour across all years in a five-day window. For instance, if I wanted to impute the wind speed at 08:00 on 23 June, I would take the average wind speed at 08:00 on 21,22,23,24,25 June for all non-missing years in the dataset. Apart from whether this is a great method, it has a "smoothing" effect so that the variance of the imputed wind speeds is much lower than the observed wind speeds.
What I want to do is add noise to the imputed data so that the variance is equal to the variance of the time series overall. I thought I could take the following approach, based on the fact that the variance of the sum of two normally distributed random variables is the sum of their two variances:
- Find the standard deviation of the logarithm of the imputed values
- Find the standard deviation of the logarithm of the observed values
- Take the square root of the difference of the squared standard deviations:
sd_diff = sqrt(sd(log(O))^2 - sd(log(I))^2)
- Generate Gaussian noise with standard deviation
noise = rnorm(length(I), mean = 0, sd = sd_diff)
- Add this noise to the logarithm of the imputed values and back-transform:
I_new = exp(log(I) + noise)
This approach, without the log transformation, worked for me to add noise to other imputed time series that were roughly normally distributed. However, when I use this approach with the log transformation, I end up with a higher standard deviation of the imputed values than of the observed values. I have the feeling that this relates to Jensen's inequality (
log(sd(x)) < sd(log(x))). Is there any way that I can adjust the standard deviation of the noise so that I can achieve my goal of having the standard deviation of the imputed values match that of the observed values?