5
$\begingroup$

In most work I've seen, MLPs (multilayer perceptron, the most typical feedforward neural network) and RBF (radial basis function) networks are compared as distinct models, where

  • MLP neuron outputs $\sigma(\mathbf{w}^\top \mathbf{x})$.
    • $\sigma$ is a nonlinearity/activation function, e.g. ReLU, sigmoid, tanh
  • RBF network neuron outputs $\phi_{k}\left(\left\|\mathbf{x}-\mathbf{c}_{k}\right\|_{2}\right)$
    • $\phi_k$ is also an activation function, but an RBF function
    • instead of dot product between inputs and weights, does euclidean distance
    • weights are interpreted as centers $\mathbf{c}_k$

I have 2 slightly-related questions:

1) Why aren't RBF activation functions $\phi(\cdot)$ ever used in MLPs, i.e.

$$\phi(\mathbf{w}^\top \mathbf{x})?$$

2) Or are RBF networks actually covered in the general MLP formulation $\sigma(\mathbf{w}^\top \mathbf{x})$ where $\sigma=\phi$, just by choosing different priors on w and b?

All the popular nonlinearities are monotonic (or if not, they are "almost-monotonic" e.g. Mish activation function, Google's Swish/SIL/SiLU, GELU). However, this answer by Toni Bellamo shows that non-monotonic activation functions work well too. Yoshuo Bengio in that same question also agrees. Furthermore, there is also a lot of research exposing fundamental flaws with the popular, monotonic ReLU-like activation functions*.

Is it so that, as these lecture notes put it,

the rarely questioned restriction to monotonic activation functions constitutes a surprisingly persistent meme that arose in the 1940s

Are there other reasons to keep functions monotonic---other than guaranteeing convexity in single-layer networks---or is this indeed mostly a "meme"?

Goodfellow tried Gaussian activation functions exp(-z^2) with both forms but I don't think he included it in the paper:

  • If z=wTx (a linear function), then the model was still highly vulnerable to adversarial examples.
  • If z=||w-x||2 (a quadratic template matching function), then the result depended on the depth of the model.
    • Shallow models worked OK (for their depth) and were noticeably resistant to adversarial examples.
    • Deep models were too difficult to train.

* See Goodfellow et al, 2015 and Hein et al, 2019

** Wu, Huaiqin (2009). "Global stability analysis of a general class of discontinuous neural networks with linear growth activation functions".


Update: Found some sinusoidal activation functions work, but still interested in RBF-like ones as well.

$\endgroup$
6
  • 1
    $\begingroup$ RBF networks do exist. apps.dtic.mil/dtic/tr/fulltext/u2/a196234.pdf I wonder if they're more challenging to train, because you have to select "good" centers for the RBF functions to make headway, and an exponentially decaying gradient implies that the vanishing gradient problem is severe. Just a guess, though. // Some of the universal approximation theorems for neural nets stipulate monotonic activation functions, so that could be one reason monotonicity persists. $\endgroup$
    – Sycorax
    May 26 '20 at 16:37
  • $\begingroup$ @Sycorax, but Broomhead & Lowe's RBF networks are not trained by backprop. They are essentially just a linear network with a nonlinear transformation of the input (much as applying powers of your input variables). $\endgroup$
    – seanv507
    May 26 '20 at 16:47
  • $\begingroup$ @seanv507 right, what I'm guessing is that the reason that RBF networks aren't widely used today is that they're challenging to train with backprop. If you use a Broomhead & Lowe network, you're not really taking advantage of all the more recent innovations in NNs (deeper networks and so on) -- as you note. The citation is there because RBF networks do exist, they're just not widely used. $\endgroup$
    – Sycorax
    May 26 '20 at 16:56
  • 1
    $\begingroup$ so I would say the reason Broomhead & Lowe networks are not widely used today is that they have basically been replaced by gaussian SVM, which essentially place a center at each data point ...(but again are not trained by backprop) $\endgroup$
    – seanv507
    May 26 '20 at 18:25
  • 1
    $\begingroup$ I already mentioned RBF networks in my original answer but I was hoping for MLPs with RBF activations, i.e. dot product w^T x instead of L2 norm ||x-w|| :) unless those two can be expressed as the same thing? $\endgroup$ May 26 '20 at 19:43
1
$\begingroup$

Seems that some people are a bit confused by your question because of its title; indeed, RBF neural networks exist, but they are a different architecture than the traditional multi-layer NNs. The body of your question is complete and formulated well (apart from some ambiguity because of the use of MLP), so to get to the answer let me ask a slightly different question: why are some activation functions more popular than others? Why are there only some activation functions in use? There is an infinite number of functions that are not used as activation functions in multi-layer NNs. One could think about some activation function that is a slight modification of the popular one, for example by squaring the argument, and ask "Why this kind of a function is not widely used as an activation function?"

In multi-layer NNs, each layer transforms the space in some (usually non-linear) way. When we use a non-monotonic activation function, we can imagine how a single layer of neurons works. But what would happen if we used another layer with a non-monotonic activation function transforming the outputs of the first layer? What if we added yet another such layer?

  • As you can imagine, the transformation of the original space gets more sophisticated and twisted more quickly compared to if we used a monotonic activation function.
  • The training process may also be more difficult because the error landscape is more complicated.
  • Calculating the derivatives may be less efficient depending on the formula of the activation function.

So it is not like you cannot use a Gaussian as an activation function in a multi-layer NN. You can use it, and you can use other unpopular activation functions, and for a particular data and a particular NN topology they may even be more efficient and yield better predictions than those provided by the most popular activation functions. The question is: Why use this function and not the other? Why is this particular activation function beneficial? How is this function better from the other one that works well in a general case, hence it became so popular?

In a similar vein, you can use different activation functions (other than the most popular Gaussian) in the RBF NN architecture, and some of them will make more sense (and will perform better) than others.

This reminds me of another question that I want to bring up here just for illustration: how many layers in a NN you should use? We know that a single hidden layer is sufficient. So why people use more hidden layers? For efficiency, because of the lower total number of neurons needed when more layers are introduced? For better generalization? This demonstrates that your goal may be achieved in many ways, and it is good to first define some evaluation criteria to be able to measure the performance of different possible approaches and compare them.

Coming back to the original question "Why aren't [traditional] neural networks used with RBF activation functions (or other non-monotonic ones)?" – They are, but if one wants to use many such layers, given the potential problems I mentioned above, one should justify this decision by stating "This activation function is better than sigmoid or ReLU because..."

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.