Training a Neural Network where each class can be linearly divided into sub-classes

Question

Suppose I want to train a neural network on some multivariate data. This data is not linearly separable, and will require a non-linear, multi-layer neural network for effective classification.

Each data vector consists of two parts. The first part represents an observation, and the second part represents the day on which that observation was made. Now, if observations of one class are made on the same day, then I know that there is some correlation between the observation data. However, if observations of the same class are made on different days, I know that there is no correlation between the observation data. Effectively, every class-day pair could be treated as a separate class, and a new classifier could be trained on these classes.

So there are two ways of training this neural network. The first method is to train a single neural network, taking in both parts of the data vector. The second method is to train multiple neural networks, one for each day, taking in only the observation part of the data vector.

My question is: Would the second method necessarily perform better than the first?

Answer 1

In general, any true fact you encode will improve the performance of your algorithm, because it won't have to waste time or possibility on things known to be wrong. It's also often the case that the sort of structure that can exist as a hypothesis is insufficient to express the sort of facts that are easy to encode. (If using datetimes instead of dates, for example, the hypothesis "days transition at midnight" might not be able to be expressed simply, but need to be $n-1$ linear separations for $n$ days, with no correlation between them making the $i$th one any more likely to be at midnight because all the previous ones were.)

The main reason why I would expect the first method to perform better would be if there is some sort of low-level structure in the data that is shared between the days, such that there is no correlation between class membership day to day but there is a correlation between the optimal weights of the early nodes in the hidden layers. Even in this case, one could still take advantage of this by sharing nodes in the early network and enforcing separation in the late network.