how to encode classes so that they reflect sequential correlation between them

I have a time series that I want to predict (using Neural Network, but is not important).

On top of using the info at previous values ($t-1$, $t-2$, ...etc) I want to use another variable: the hour of the day.

Stupid Method

One way would be to map:

$00:00 \rightarrow 0$

$01:00 \rightarrow 1$

.... and so on. So I woul have $24$ classes $\{0,1,2,..., 23\}$. However there is a problem: $23$ and $0$ are close, but at the same time this is not reflected here.

second method: one hot encoding?

Another way would be to create vectors with $24$ entries where all the elements are zero, apart from the class that we are looking at. This is equivalent to creating dummy variables. For instance:

$00:00 \rightarrow [1,0,0,0,....,0]$

$01:00 \rightarrow [0,1,0,0,...,0]$

...and so on. However there is a problem: this doesn not reflect correlation at all!

Other Method

I was thinking of using some cyclic structure, for instance $sin$, $cos$ or maybe polar coordinates?

I haven't found the solution yet, but this is my progress:

1. Map the class labels of the stupid method to $[0,1]$, by normalizing them
2. Scale them to be in the range from $0$ to $2\pi$.
3. Use $cos$ or $sin$ on them.

This method doesn't quite work, but I can see some light.. Any ideas?

• I have an idea! If we have say $6$ classes, $\{0,1,2,3,4,5}$ we could create 5 features instead of 1. Basically we keep on moving them up to one. so that we obtain the following: array([[ 0, 15, 24, 33, 42, 51], [ 1, 10, 25, 34, 43, 52], [ 2, 11, 20, 35, 44, 53], [ 3, 12, 21, 30, 45, 54], [ 4, 13, 22, 31, 40, 55], [ 5, 14, 23, 32, 41, 50]]) Oct 9 '17 at 14:52
• Basically the first column is [0,1,2,3,4,5] , the second column still goes from 10 to 15 but it is shifted! So that the "hole" happens somewhere else, so it is [15,10,11,12,13,14] then the next column is shifted again so [24,25,20,21,22,23] and so on. Then to compare the classes you can take the difference vector and calculate its norm. Indeed you can take the difference between the zero row of this matrix and the first (or the last) row, take the norm and you will get the same result!! Oct 9 '17 at 14:54

What you're basically trying to do is convert your dataset from a time-like structure to a feature-like one. I once tried to do the same with random forest, i.e. add extra features on top of lagged terms as inputs. I tried several different things, notably ways of detecting seasonality, cyclicity, and trend, and then transforming them into features (I was successful only to some extent).

With neural networks, however, you don't need to do any of those things. As described in this book chapter (and as I have experienced myself), neural network autoregression works quite well without adding any extra time-based features on top of lagged variables. As explained in section "neural network autoregression", this method can be efficient without any time-based features because it emulates a nonparametric $AR(p)$ model (or a $ARIMA(p,0,0)$), therefore the time-dependent information is extracted from the lagged terms alone. For a network of $p$ lagged input variables, $m$ hidden neurons in a single hidden layer, and the identity activation function $g(x) = x$, you get a model like this:

$$\hat y_t = w_0 + (\sum_i^m w_i ) \sum_k^p w_k y_{t-k}$$

Which is, well, equivalent to an $AR(p)$ or $ARIMA(p,0,0)$ process of the form:

$$\hat y_t = c+ \sum_k^p \phi_k y_{t-k}$$

I should probably also point out that the implementation described in the book applies forecast aggregation (trains multiple networks and averages their prediction). If you want something more "advanced" with a single neural network, you should look into LSTM Networks.

With your methods you're increasing computational complexity without any guaranteed payoff, and that's never a good thing.

• thank you for your answer! I am quite new to neural networks and machine learning in general, that is probably why my methods are so wrong! Anyway, I am using a MLP with MLPRegressor in sklearn (before getting my hands dirty with RNN and in particular LSTM), however no matter what architecture/functions I try, but the RMSE on the test set, when I try to predict $t+1$ never goes below 595. So I thought new features could help, features such as the time Oct 9 '17 at 15:40
• Do you have a "guess" or an intuition of why/how the neural network would work well without giving it this extra feature? Oct 9 '17 at 15:41
• Also, can you tell me more or less where it is described in the paper you linked? Oct 9 '17 at 15:58
• @Euler_Salter, I tried answering all of your queries at once by updating my answer. Oct 10 '17 at 7:18