I need to add an hour feature to my data.

Assigning "dummy variables" (which is what I usually do) does not work in this case, apparently (wouldn't elaborate - but believe me this is so).

At first, I simply assigned the hour itself (13:00 - 13, 14:00 - 14, etc...)

But then I realized this means 23:00=23 is radically different in value from 00:00=0, which is clearly undesirable since they are quite close in reality.

I read in google and found there's the following transformation: sin(2*pi / 24 * hour) which would result in hour 23:00 and hour 00:00 both giving close values.

However, I have several issues with the new method:

  1. hours 5 and 7 (for instance) are indistinguishable to the algorithm as they have the same value (0.965)
  2. Another problem is that, for instance, the difference in value between hour 6 (1.0) and 7 (0.965) is different from the different of 7 (0.965) to 8 (0.877) - which, of course, does not reflect reality (at least not in my case).

Is there any other, better solution to this problem than I raise above?

  • $\begingroup$ Why is closeness in time an issue in modelling hours for your model? $\endgroup$ – Michael R. Chernick Jun 21 '17 at 17:56

A linear time effect can be used to model certain phenomena, like the accumulation of muscle relaxants as participants endure longer periods of forced wakefulness, or growth trends of alligators. In general, though, it is a bad idea to use linear time for the reasons you mention, since when those assumptions fail, they can fail catastrophically. However, those assumptions can be checked by using a residual plot which you should do.

The "google" source is a start but likely not a good suggestion unless you can better describe the problem. it is just modeling something that "goes up" at hours 6 and 18 and "goes down" at hours 0 and 12. Whether or not that's appropriate is your call and yours alone. You have described some limitations which I can address directly in a proposed solution below, but rather than discarding ideas which are or are not applicable to your problem, try describing it.

You can use a complex representation of cyclic time using convolutions or equivalently splines. This uses multiple features, but you can subselect sparse representations with any feature selection method.

A continuous linear representation of time involves creating a triangle wave representation of time. Using 6 hour intervals, a time data frame would map to the following 4 features:

enter image description here

Which, demonstrating with regression creates the following:

rep <- bs(0:23, degree = 1, knots=c(1:3*6))
rep[1:6, 4] <- rev(rep[19:24, 4])
y <- rnorm(24, sin(0:23 * pi / 12), 0.4)
fit <- lm(y ~ 0 + rep)

enter image description here

It should be said this is basically a poor man's fast-fourier transform. If you fit a regression model, you can use FFT to handle cyclic auto-regressive or time-depend errors by using a frequency decomposition to estimate what fluctuations in errors owe to unknown periodic effects. This is discussed in some detail in Diggle's Book Time Series: A biostatistical introduction, which might be a good reference in general.

| cite | improve this answer | |

Not sure what exactly your goal is here, maybe then there may be a better solution. However, if you only want to disentangle the hours 5 and 7 (among others) the simplest solution is to create two features:

  • sin(2*pi / 24 * hour)
  • cos(2*pi / 24 * hour)

Which basically translates each our to its position on the clock circle. Depending on the application (e.g. if you are doing some kind of clustering), this may just work. For other applications, this may fail horribly, but it is impossible to tell with this little information.

| cite | improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.