I have high-frequency time series (observations every few minutes) for which I wish to compute daily averages. The data exhibit a strong diel cycle. Sometimes observations are missing in the time series.

For example, I could have 50% of the data missing on a given day.

The effect of this missingness on the daily average isn't immediately obvious given the proportion of missing data. Evenly spaced missingness has little affect (say that 5-minute and 10-minute data give ~same daily average), whereas if all of the missing data are concentrated around the daily minimum/maximum then the daily mean would be biased.

However, the effect of missingness on the daily average not only dependent on the percent of data missing, but also on the evenness of that missingness.

Does anyone know of a good approach for characterizing the potential for missingness to bias the daily average? I'm envisioning some sort of index that incorporates the extent and temporal distribution of missing values, and other factors I may not be considering. Thanks for any guidance.

The figure and R code below use a simple sine wave to illustrate my thinking:

enter image description here

# ===================================
# = Series w/ complete observations =
# ===================================
X <- (1:288)*pi*2*(1/288)
sX <- sin(X)

# =================================================
# = Series w/ 50% missing, but evenly distributed =
# =================================================
X1 <- seq(2, 288, 2)*pi*2*(1/288)
sX1 <- sin(X1)

# =================================
# = Series w/ 50% missing, random =
# =================================
iX2 <- sort(sample(1:288, 144))
X2 <- X
X2[iX2] <- NA
sX2 <- sin(X2)

# ==================================
# = Series w/ 50% missing, clumped =
# ==================================
iX3 <- sort(sample(1:288, 144, prob=1.5^(X+1)))
X3 <- X
X3[iX3] <- NA
sX3 <- sin(X3)

# =====================================
# = Graph time series and their means =
# =====================================
dev.new(height=7, width=3.5)
par(mfrow=c(4,1), mar=c(2,2,0.5,0.5), ps=9, cex=1, mgp=c(1,0.25,0), tcl=-0.25)
legend2 <- function(s, lab){legend("topright", legend=paste(lab, "mean =", round(mean(s, na.rm=TRUE),1)))}

legend2(sX, "100%,")

legend2(sX1, "50%, even,")

legend2(sX2, "50%, random,")

legend2(sX3, "50%, clumped,")

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.