16
$\begingroup$

So when looking at Radial Basis Function Neural Networks, I've noticed that people only ever recommend the usage of 1 hidden layer, whereas with multilayer perceptron neural networks more layers is considered better.

Given that RBF networks can be trained with version of back propagation is there any reasons why deeper RBF networks wouldn't work, or that an RBF layer couldn't be used as the penultimate or first layer in a deep MLP network? (I was thinking the penultimate layer so it could essentially be trained on the features learned by the previous MLP layers)

Improve this question
$\endgroup$
1
  • 2
    $\begingroup$ That was in the earlier days of NN research, however now more layers is typically the recipe for greater performance (deep learning). I think the current favourite approach is a smart initialisation, as many layers as possible, regularisation via dropout and softmax instead of sigmoidal activations to avoid saturation. (But I may be wrong on the techniques). I think some people also use iterative deepening to get better results. Also, Google got state of the art on imageNet in 2014 with a 100 layer network. $\endgroup$
    – user1646196
    Commented May 11, 2015 at 10:37

1 Answer 1

Reset to default
5
$\begingroup$

The fundamental problem is that RBFs are a) too nonlinear, b) do not do dimension reduction.

because of a) RBFs were always trained by k-means rather than gradient descent.

I would claim that the main success in Deep NNs is conv nets, where one of the key parts is dimension reduction: although working with say 128x128x3=50,000 inputs, each neuron has a restricted receptive field, and there are much fewer neurons in each layer.In a given layer in an MLP- each neuron represents a feature/dimension) so you are constantly reducing dimensionality (in going from layer to layer).

Although one could make the RBF covariance matrix adaptive and so do dimension reduction, this makes it even harder to train.

Improve this answer
$\endgroup$
3
  • $\begingroup$ I recently read a paper that proposed a back-propagation algorithm for training RBF networks. Given this could their be a benefit in having an RBF as the final layer in a deep network? I suppose in this form the rest of the deep network would essentially be detecting features that the RBF can classify $\endgroup$
    – user1646196
    Commented May 12, 2015 at 19:35
  • $\begingroup$ maybe you should link to the paper and then people can give more informed answers. I don't see any benefit...given that the RBF is too non linear (and eg sigmoids have been replaced by relu because they were too non linear- vanishing gradient...). What people do is train with conv net with standard mlp on top, then throw away mlp and use svm $\endgroup$
    – seanv507
    Commented May 13, 2015 at 7:26
  • $\begingroup$ The paper is "Training RBF networks with selective backpropagation" not sure if you can read it here or if there's a paywall sciencedirect.com/science/article/pii/S0925231203005411 . I wasn't aware sigmoids had been replaced by relu because of non-linearity, but given that I can see how increased non-linearity would be shied away from. I'll mark the answer as accepted :) $\endgroup$
    – user1646196
    Commented May 13, 2015 at 8:42

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.