Background
I'm working on a project which aims to use the history data about a water flux to detect whether there is a leakage happened. The data is hourly collected and among about 4 months.
I've already read the book which Professor Hyndman write about the forecast and some posts about outliers/anomaly detection on the site, but still I get confused how to realize this in R. In the meantime, I think I've got things mixed up and want to know the basic procedure to accomplish it.
What I've tried
At first, I think the basic idea is to fit a model on my train data and forecast it with the test part. Then use the model to check the residual in the whole data whether they are all normal distributed or at least has zero mean.
So according to Timeseries analysis procedure and methods using R, I've tried ARIMA, Exponential Soomthing and TBATS, but the result isn't ideal. And I'm also afraid that this could lead to a flaw since I didn't consider the outliers and anomaly.
Here is my code
model <- list(
mod_arima <- auto.arima(train_h, ic = "aic"),
mod_exp <- ets(train_h, ic = "aic"),
mod_tbats <- tbats(train_h,ic = "aic")
)
forecasts <- lapply(model, forecast, h = 24)
par(mfrow = c(2,2));
for (i in forecasts) {plot(i); lines(test_h,col = "red")}
Then according to Simple algorithm for online outlier detection of a generic time series, I find I could detect those single point that in my data through the answer by professor Hyndman, but I fail to change to detect the small level shift. (I've tried to create a 0.05*mean shift level, removing the outliers, then using the tso to detech the level shift, however it fail totally...)
My Problems
My problems mainly falls in the following two parts:
Even though it seems that there is a relativity between the flux and the flux an hour ago(Looking from the plot), could I use the hourly data directly to fit a model or should I first select the data at the same time each day to fit a model each?
Now I think my problem could be partly solved by directly detecting the level shift in the data, but I think that the leakage in the flux data should be relatively small if any(maybe just 5%,10% of the mean). While I've mannually create a shift in a try, when I use the tsoutliers::tso in R directly, the result isn't ideal. Is this idea right or should I fit a model still? And how could I detect such a small change in the level shift in a time series, particularly in R?
ps:Since I'm new to Cross Validated, I fail to find a way to upload the data may be easy for you solve my problem, is there any advice?