Removing seasonality from a dataset where each 24 hour period of a day is normally or bimodally distributed

Question

I'm trying to understand how best to approach the hypothetical example below.

If have a data series which represents the load time of an web application and the load time increases as more people use the app.

I have hourly average data going back x number of months. The dataset shows a weekly trend (Fridays appear similar to other Fridays) and an hourly trend (8pm looks similar to 8pm the day prior).

Assume that the hourly data has a bimodal or normal distribution. I'm keen to understand how this impacts the thought process.

My end goal it to remove the seasonality added by the peaks and troughs that the additional users create to see the load times independent of the users. I.e. when in load time small or large for another factor.

When i've been thinking about this and doing some digging (i have limited statistical knowledge at this point) i have considered differencing on an hourly basis (comparing Wednesday 8pm to the prior Wednesday at 8pm) due to a few online examples with weather data that seemed in some respect to map.

I'd really like to understand how someone would approach this problem and the thought processes they would go through to get the desired outcome.

Thanks

Answer 1

I find it hard to model a hourly data, periodic in its nature, to follow a normal distribution.

Regarding the periodic effects, you may try to use sine and cosine functions for each period, assuming that you already know the periods. $$ x(t) = \sum_j a_j \sin (\omega_j t) + b_j \cos (\omega_j t) \ , $$ where the angular frequencies $\omega_j = 2 \pi / T_j$ are related with each period $T_j$. With this values, we can compute the amplitude $A_j=\sqrt{a_j^2+b_j^2}$ and the phase $\phi_j = \arctan \frac{b_j}{a_j}$ of each periodic effect.

Answer 2

This is a fairly common problem in time series analysis (multiple deterministic seasonalities) where observations are possibly/probably affected by deterministic factors such as hour-of-the-day , day-of-the-week, day-of-the-month , week-of-the-month, holiday-effects , particular months-of-the-year , level shifts , local time trends and memory affects (arima) in addition to user-specified predictor series like temperature or price.

https://autobox.com/pdfs/SARMAX.pdf is the goal where the X's and the I's which are latent in the data can be identified and used to characterize the data for either forecasting or early-warning detection.

Models like this are quite understandable and are quite flexible

I have seen successful applications involving TACO-BELL 15-minute forecasts and call center-forecasts for HP https://demand-planning.com/2010/03/18/can-forecasting-help-me-staff-a-specific-hewlett-packard-call-center-at-1030-am-on-a-friday/ and a number of power consumption studies.

Following are a couple of my posts on this subject. Forecasting data with multiple seasonality

and

Hidden markov model to detect Stock outs in Hourly sales Time series data

and

What model to fit given ACF and PACF (seasonal data)

I would stay very clear of suggestions about fitting arima models with frequencies of 24x7=168 etc. as they miss the target by attempting to use memory as compared to identifying logical causal variables that can be found in the data.

Finally the concern should be about the distribution of residuals from a model NOT the distribution of the original data as the distribution of errors is where the assumptions are placed. With just a few months of data it will be impossible to pick up a number of the possible predictors that I mentioned BUT maybe this response will motivate you to find more data .

With just a few months of data , the best you might hope for are daily effects , hourly effects , possible anomalies and possibly level shifts.

Perhaps your question should be "Incorporating" rather than "Removing" with both ending up in the same analytical place.

Answer 3

Before attempting to model the seasonal effect, I would use a simple t-test to confirm the existence of a seasonal effect. Divide the observations into subsets and check the subset for significant difference from the total dataset of observations.
The null hypothesis is the observations in the subset have the same mean as a random sample from the total dataset. The alternative hypothesis is the subset has a different mean than a random sample from the total dataset.
From the description, it is obvious that the subsets "Fridays" or "8pm" are significantly different from the total dataset. If the subset "January" is not significantly different from the total dataset, then maybe there is no value in attempting to include seasonal effects in the model.