3
$\begingroup$

I have a data set with hourly values over two years. The first year has 366 days the second has 365 days. There is a clear monthly pattern of the data, as the values are higher in the winter as in the summer. There is a clear weekly pattern, as the values are higher during the week than at weekends. And there is a clear hourly pattern, as the values are higher during the day compared to the night. Assuming I have the variable y and all the data for y. This variable has a stochastic part z and a seasonal part s. So y = z + s, where s = a0 + a1*h + a2*d + a3*m. In this case a1 is the coefficient for the hourly data, which means a1 comprises 24 subcoefficients. For instance a1(10) = 0.25 and a1(5) = -0.1. Which means that the hour 10 has an impact of 0.25 on the data and the hour 5 (in the night) has a slightly negativ effect on the data. a2 is the coefficient for the weekdays and should have 7 subcoefficients. a3 stands for the particular month and should have 12 subcoefficients. The idea is to take the first element as a reference. For the hourly data it means I take the hour 1 as a reference and estimate the effects of the other 23 hours in an OLS regression. Do you have any clue how to estimate these coefficients? I read in the literature that a step function for s with dummy variables might be helpful. I don't know exactly what it means. I checked out different functions to estimate or smooth seasonal effects (filter, ma.filter, decompose, bats). But I don't like them as they do not give me the clear estimates of the coefficients and somehow they do not give me the smoothing effects I expected (They did not capture the difference between summer and winter).
The detrended data looks like this:

df[1:10,1:2]

                 time      y
1  2016-01-01 00:00:00  -2.907587
2  2016-01-01 01:00:00  -4.378144
3  2016-01-01 02:00:00  -6.178701
4  2016-01-01 03:00:00  -9.959259
5  2016-01-01 04:00:00  -9.359816
6  2016-01-01 05:00:00  -9.750373
7  2016-01-01 06:00:00 -10.910930
8  2016-01-01 07:00:00  -8.611487
9  2016-01-01 08:00:00  -9.042045
10 2016-01-01 09:00:00  -7.002602

 str(df[,1:2])
'data.frame':   17544 obs. of  2 variables:
 $ time : POSIXct, format: "2016-01-01 00:00:00" "2016-01-01 01:00:00" 
 "2016-01-01 02:00:00" ...
 $ y: num  -2.91 -4.38 -6.18 -9.96 -9.36 ...

I have got the idea from this paper (Formula (1)): https://ac.els-cdn.com/S0140988314000875/1-s2.0-S0140988314000875-main.pdf?_tid=5cd1edbe-2441-40db-8a1f-65bee19cd4d0&acdnat=1521371765_675fe9c00865a4882ed733e03b908618

$\endgroup$

migrated from stackoverflow.com Mar 18 '18 at 17:25

This question came from our site for professional and enthusiast programmers.

  • $\begingroup$ To those voting to move the question: perhaps the question formulation is too long, but ultimately it simply asks how to programatically include particular variables. Hence, it is perfectly on-topic. $\endgroup$ – Julius Vainora Mar 18 '18 at 17:10
2
$\begingroup$

Your example data is not really enough to fully verify the code, so let

times <- seq(ISOdate(2016, 1, 1, 0, 0, 0), ISOdate(2018, 1, 1, 0, 0, 0), "hour")
y <- rnorm(length(times))

Now let's construct the other variables:

data <- data.frame(date = times, month = factor(months(times)), 
                   day = factor(weekdays(times)), 
                   hour = factor(format(times, "%H")), 
                   y = y)
head(data, 2)
#                  date   month    day hour           y
# 1 2016-01-01 00:00:00 January Friday   00 -0.04493361
# 2 2016-01-01 01:00:00 January Friday   01 -0.01619026

That was the hard part because the result is

lm(y ~ month + day + hour, data = data)
# 
# Call:
# lm(formula = y ~ month + day + hour, data = data)
# 
# Coefficients:
#    (Intercept)     monthAugust   monthDecember   monthFebruary    monthJanuary  
#       0.040847       -0.005865       -0.027025       -0.016836        0.023103  
#      monthJuly       monthJune      monthMarch        monthMay   monthNovember  
#      -0.042943        0.026567       -0.014632        0.048680       -0.031783  
#   monthOctober  monthSeptember       dayMonday     daySaturday       daySunday  
#      -0.035234        0.014624       -0.032229       -0.001063       -0.013470  
#    dayThursday      dayTuesday    dayWednesday          hour01          hour02  
#      -0.026101        0.013825       -0.045751       -0.045184        0.029389  
#         hour03          hour04          hour05          hour06          hour07  
#      -0.015768       -0.043531       -0.043825       -0.016781       -0.011752  
#         hour08          hour09          hour10          hour11          hour12  
#      -0.025495       -0.076518        0.003449        0.026689        0.010491  
#         hour13          hour14          hour15          hour16          hour17  
#       0.024666       -0.079720        0.035006        0.077219       -0.022182  
#         hour18          hour19          hour20          hour21          hour22  
#      -0.008761       -0.017968       -0.115603        0.041685        0.052371  
#         hour23  
#      -0.073574  

We see that variables corresponding to April, Friday, and 00 hour are omitted. That's the reference (not just some hour). You could change it by specifying levels of month, day, and hour (currently it is set alphabetically).

$\endgroup$
  • $\begingroup$ Oh. Yes. That is what I was looking for. Thanks. So the trick is to write the dependent variables into factors and the regression gives a factorized output. Great. If there is no reference I guess the reference is the mean of y. But actually I don't see a difference in adding -1 or not? The only thing I see is that in both cases one hour is missing -> hour00 $\endgroup$ – Carolus Fridericus Mar 18 '18 at 16:37
  • $\begingroup$ So is the reference to all coefficients then hour00? $\endgroup$ – Carolus Fridericus Mar 18 '18 at 16:38
  • $\begingroup$ sry. *independent variables $\endgroup$ – Carolus Fridericus Mar 18 '18 at 16:53
  • 1
    $\begingroup$ @CarolusFridericus, see the update, didn't think about the reference myself. There's no point really to exclude the intercept in this case. Since we include all the levels of three categorical variables (not just hour), now the reference is more complex, not just hour00. $\endgroup$ – Julius Vainora Mar 18 '18 at 17:29
  • 1
    $\begingroup$ @CarolusFridericus, I don't really understand what you are trying to achieve by multiplying those coefficients by the values of y. It's most likely wrong, because those coefficients are additive effects of, say, a month being March. As for the reference, I think what you are trying to achieve also matters very much. Since those are more methodological issues, no longer just estimation, I suggest to post another question here, in stats.se, and you will surely get some useful answers. $\endgroup$ – Julius Vainora Mar 18 '18 at 22:44

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.