2
$\begingroup$

I'm trying to understand how best to approach the hypothetical example below.

If have a data series which represents the load time of an web application and the load time increases as more people use the app.

I have hourly average data going back x number of months. The dataset shows a weekly trend (Fridays appear similar to other Fridays) and an hourly trend (8pm looks similar to 8pm the day prior).

Assume that the hourly data has a bimodal or normal distribution. I'm keen to understand how this impacts the thought process.

My end goal it to remove the seasonality added by the peaks and troughs that the additional users create to see the load times independent of the users. I.e. when in load time small or large for another factor.

When i've been thinking about this and doing some digging (i have limited statistical knowledge at this point) i have considered differencing on an hourly basis (comparing Wednesday 8pm to the prior Wednesday at 8pm) due to a few online examples with weather data that seemed in some respect to map.

I'd really like to understand how someone would approach this problem and the thought processes they would go through to get the desired outcome.

Thanks

$\endgroup$
0
$\begingroup$

I find it hard to model a hourly data, periodic in its nature, to follow a normal distribution.

Regarding the periodic effects, you may try to use sine and cosine functions for each period, assuming that you already know the periods. $$ x(t) = \sum_j a_j \sin (\omega_j t) + b_j \cos (\omega_j t) \ , $$ where the angular frequencies $\omega_j = 2 \pi / T_j$ are related with each period $T_j$. With this values, we can compute the amplitude $A_j=\sqrt{a_j^2+b_j^2}$ and the phase $\phi_j = \arctan \frac{b_j}{a_j}$ of each periodic effect.

$\endgroup$
0
$\begingroup$

This is a fairly common problem in time series analysis (multiple deterministic seasonalities) where observations are possibly/probably affected by deterministic factors such as hour-of-the-day , day-of-the-week, day-of-the-month , week-of-the-month, holiday-effects , particular months-of-the-year , level shifts , local time trends and memory affects (arima) in addition to user-specified predictor series like temperature or price.

https://autobox.com/pdfs/SARMAX.pdf is the goal where the X's and the I's which are latent in the data can be identified and used to characterize the data for either forecasting or early-warning detection.

Models like this are quite understandable and are quite flexible

I have seen successful applications involving TACO-BELL 15-minute forecasts and call center-forecasts for HP https://demand-planning.com/2010/03/18/can-forecasting-help-me-staff-a-specific-hewlett-packard-call-center-at-1030-am-on-a-friday/ and a number of power consumption studies.

Following are a couple of my posts on this subject. Forecasting data with multiple seasonality

and

Hidden markov model to detect Stock outs in Hourly sales Time series data

and

What model to fit given ACF and PACF (seasonal data)

I would stay very clear of suggestions about fitting arima models with frequencies of 24x7=168 etc. as they miss the target by attempting to use memory as compared to identifying logical causal variables that can be found in the data.

Finally the concern should be about the distribution of residuals from a model NOT the distribution of the original data as the distribution of errors is where the assumptions are placed. With just a few months of data it will be impossible to pick up a number of the possible predictors that I mentioned BUT maybe this response will motivate you to find more data .

With just a few months of data , the best you might hope for are daily effects , hourly effects , possible anomalies and possibly level shifts.

Perhaps your question should be "Incorporating" rather than "Removing" with both ending up in the same analytical place.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.