3
$\begingroup$

I am trying to understand the interpretation of radial basis functions (RBFs) as networks and then trying to understand the relationship it has to "normal" neural networks and how to extend them to multiple layers.

For this I was watching the lectures from the caltech course CS156 and they present the following slide:

enter image description here

This slide shows that RBF networks and neural networks are quite different and extending the RBF network to multiple laters is not 100% clear to me how it would be done (or if it even makes sense to do).

One way that I thought to extend RBFs to multiple layers is by having a general neural network as a usual neural network, but then substitute each activation function with a Gaussian function.

However, I thought of a different way of doing it more similar to the way the RBF network is build but extend that recursively. The main idea is to make sure each later computes feature based on the distance to a center with respect to the previous layer. Something of the form:

$$ x^{(l)}_k = \phi( \| x^{(l-1)} - \mu^{l}_k \| )$$

where $x^{(l)}_k$ denotes the output of the activation of a certain activation node in the neural network. The idea is that its getting the whole output of the previous later, computing a distance with respect to a center of the current hidden layer and then applying an non-linearity to it. To illustrate my idea, here how the first later would look:

enter image description here

For a arbitrary layer:

enter image description here

The issue with such a network is that even though it makes sense as a generalization to the RBF netowrk, its structure is completely different from neural networks since only the last layer has weights to be updated. Another issue is choosing the centers. The centers could just be chosen with k-means at each layer but I am not 100% if this is justifiable in any framework (like regularization).

Anyway, my main question is, how are RBF networks extended to use multiple lyaers like the multiple layer perceptron (MLP/neural networks) are used. I provided two suggestion but wasn't sure which was correct or which one was actually used or which one actually had some research done.

$\endgroup$
  • 1
    $\begingroup$ @404pio why dod you think multiple layer RBF doesn't make sense? $\endgroup$ – Charlie Parker Aug 11 '16 at 2:14
  • 2
    $\begingroup$ @404pio so what if they only have one layer in those books? Do they provide a reason to why not to use RBFs in multiple layers? $\endgroup$ – Charlie Parker May 16 '17 at 19:50
  • 1
    $\begingroup$ ieee.cz/knihovna/Zhang/Zhang100-ch03.pdf take a look here, maybe explanation on page 81 is sufficient? $\endgroup$ – 404pio May 18 '17 at 8:29
  • 1
    $\begingroup$ @404pio I guess the bottom line that they argue is that since RBFs are only active if the data points are close to its centers, it uses localized activations to produce approximations so it there are lots of dimensions one might need an exponential number of units. Which intuitively makes sense, but I am still not convinced, I always thought that one needed an exponential bound on the number of units for the universal approximation theorem for Neural Nets, does this just say RBFs need a even worse doubly exponential bound or something? The curse of dim seems to apply to both not just RBFs. $\endgroup$ – Charlie Parker May 18 '17 at 23:00
  • 1
    $\begingroup$ (It seems intuitively to me that they would make learning much faster, since they have only local action, making the optimization energy landscape "funnel-like", while ReLU weights have action to infinity, for instance, and make everything interdependent and the energy landscape has lots of independent branches, but I'm only thinking of single-layer 1D curve fitting, so I may be completely wrong when it becomes highly-dimensional and multi-layer.) $\endgroup$ – endolith Dec 4 '18 at 16:51

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.