Step-by-step construction of an RBF neural network

Question

I would like to solve the following task as an exercise: Given is the data in the image below. Each output $Y_j$ of the network is defined as:

$(*)$ $Y_j = \sum_{i}^N w_{ij}* exp(-\frac{||x-\mu_i||^2}{2\sigma_i^2})$

for the -ith neuron. The task is to draw an RBF network that perfectly classifies the data, with suitable means, covariances and weights. In a second step a point has to be taken and classified in the worked-out model.

My ideas We have two nodes in the input layer, one for each dimension. The hidden-layer has 4 nodes, as there are four clusters. Each of these calculates the activation given by the exponential above. The output layer has three output nodes as there are three classes. I would determine $\Sigma$ and $\mu$ as follows: For each cluster and it's datapoints $x_i$, $\mu_j$ is just the centroid of the training samples, $\sigma = \frac{1}{m}\sum_i ||x_i-\mu||$, and $\Sigma = \sigma*I_d$. I would set all weights to the correct class to 1, and everything else to zero for perfect classification.

Questions:

• If we look at the architecture of the RBF neural network, what are the weights going into the hidden layers? Usually, we sum over the inputs and weights to hidden layers $\sum x_i w_i$. However it seems that for the RBF-activation function, no such summation is needed
• Where are the means of the clusters represented in the RBF architecture? Are the means actually represented in the weights to the hidden layer, and we do not sum up input and these weights but instead subtract them, i.e calculate for each dimension of the input its distance to the respective dimension of the mean?

Answer 1

As per my best knowledge RBFs have only one hidden layer.
For the first question:
You need to weight and sum the inputs before passing through the activation. You also need to weight and sum the results of the RBFs. It should work like this:
Inputs->Sum(Inputs * Input weights)->Pass through activation function->Hidden values->Sum(Hidden values * Output parameter + bias * bias weight)->Output
Notes: Inputs: you have 2
Outputs: 3 categories
Number of RBFs: freely chosen, the more will learn more. Let's take 4 as you suggest.
Input weights: number of inputs * number of RBFs (2*4=8) because all inputs are connected to all RBFs
Hidden values: equal to the number of RBFs, (4)
Output parameters: Number of RBFs * Outputs (4*3=12) because all RBFs are connected to all outputs
Bias: always 1 with value 1
Bias weights: number of outputs (3), one per output

For the second question:
If you understand the answer on the first question then you already realized that the means are not stored in the RBF network. What perhaps is the closest is the center of the RBFs. Think about it like that your weighted and summed inputs go to each RBF. However each RBF has a different center and some will give a higher "score" to certain values and a low "score" to others. This is where differentiation happens. If you use more than 4 RBFs then this process becomes more sophisticated and accurate on the "borders" among categories. It could be said that the network understands your data this stage. Then come the output weights that influences which RBF is important for which category.

It feels a bit that you think in line with K-means clustering. Forget it, it has nothing to do with RBFs. (K-means works with the means and the distance.) Cheers from Basel!