Well, I'm still interested in a guideline or rule of thumb regarding: Given $n$ samples $\left(x, y\right)$, how to choose the hidden layers of a regression neural network? Proposals, comments and answers are highly welcome!
Nevertheless, in my question, I stated a particular situation. Despite of its exemplary nature, I think that the choice of the hidden layers should be approached from the following point of view: The non-linearity in the relation $x \to y$ can be captured through the combination of two concepts:
- univariate Polynomials (e.g. $w_m x^m + \dots + w_0 x^0$)
- "and"-junction of features (like $x_i x_k$)
Let me drop two side notes:
- Yes, the combination of these two concepts is called multivariate polynomials, but for simplicity, I didn't want to deal with them here.
- I think, it's a legitimate question: How do we actually know that these two concepts are involved in the unknown mechanism, which generates $y$ from $x$? Well, we don't know that. But we can guess. Maybe kernel-PCA will tell us.
In conclusion, I implemented "and"-layers and polynomial layers. I was using the Lasagne framework, along with scikit-neuralnetwork.
implementation
The polynomial layer maps each input feature, that is a neuron from the previous layer, to $m+1$ of its own neurons. These neurons are $w_m x^m + \dots + w_0 x^0$.
class PolynomialLayer(lasagne.layers.Layer):
def __init__(self, incoming, deg, **kwargs):
super(PolynomialLayer, self).__init__(incoming, **kwargs)
self.deg = deg
def get_output_for(self, input, **kwargs):
monomials = [input ** i for i in range(self.deg + 1)]
return T.concatenate(monomials, axis=1)
def get_output_shape_for(self, input_shape):
return (input_shape[0], input_shape[1] * (self.deg + 1))
The "and"-layer links all distinct, unordered pairs of features from the previous layer into neurons, plus it repeats each feature from that layer.
class AndLayer(lasagne.layers.Layer):
def __init__(self, incoming, **kwargs):
super(AndLayer, self).__init__(incoming, **kwargs)
def get_output_for(self, input, **kwargs):
results = [input]
for i in range(self.input_shape[1]):
for k in range(i + 1, self.input_shape[1]):
results.append((input[:, i] * input[:, k]).dimshuffle((0, 'x')))
return T.concatenate(results, axis=1)
def get_output_shape_for(self, input_shape):
return (input_shape[0], ((input_shape[1] ** 2) + input_shape[1]) / 2)
evaluation
Another question arises: In which order should be put the two hidden layers?
first network
This is the network layout:
input layer | polyn. layer | "and" layer
-------------|--------------|-------------
4 neurons | 12 neurons | 78 neurons
And here is the overwhelming result:
The learning algorithm in use was stochastic gradient descent (SGD) with a learning rate of $2 \cdot 10^{-3}$. The SGD algorithm is the frameworks' default.
second network
The alternative network layout, where we swapped the hidden layers, looks as follows:
input layer | "and" layer | polyn. layer
-------------|-------------|--------------
4 neurons | 14 neurons | 42 neurons
With the same learning rate as before, the training failed for this layout and resulted in a root mean square (RMS) error of about $8.3 \cdot 10^9$. After reducing the learning rate to $1 \cdot 10^{-3}$, at least the SGD algorithm converged properly:
Though, the RMS error is much higher, than it was with the first layout. This suggests that, despite its lower complexity in terms of neurons counts, the second layout is somehow more sensitive to the learning rate parameter. I'm still wondering, where this comes from: Explanations are highly welcome! Might it be related to the nature of back propagation?
