In most work I've seen, MLPs (multilayer perceptron, the most typical feedforward neural network) and RBF (radial basis function) networks are compared as distinct models, where

  • MLP neuron outputs $\sigma(\mathbf{w}^\top \mathbf{x})$.
    • $\sigma$ is a nonlinearity/activation function, e.g. ReLU, sigmoid, tanh
  • RBF network neuron outputs $\phi_{k}\left(\left\|\mathbf{x}-\mathbf{c}_{k}\right\|_{2}\right)$
    • $\phi_k$ is also an activation function, but an RBF function
    • instead of dot product between inputs and weights, does euclidean distance
    • weights are interpreted as centers $\mathbf{c}_k$

I have 2 slightly-related questions:

1) Why aren't RBF activation functions $\phi(\cdot)$ ever used in MLPs, i.e.

$$\phi(\mathbf{w}^\top \mathbf{x})?$$

2) Or are RBF networks actually covered in the general MLP formulation $\sigma(\mathbf{w}^\top \mathbf{x})$ where $\sigma=\phi$, just by choosing different priors on w and b?

All the popular nonlinearities are monotonic (or if not, they are "almost-monotonic" e.g. Mish activation function, Google's Swish/SIL/SiLU, GELU). However, this answer by Toni Bellamo shows that non-monotonic activation functions work well too. Yoshuo Bengio in that same question also agrees. Furthermore, there is also a lot of research exposing fundamental flaws with the popular, monotonic ReLU-like activation functions*.

Is it so that, as these lecture notes put it,

the rarely questioned restriction to monotonic activation functions constitutes a surprisingly persistent meme that arose in the 1940s

Are there other reasons to keep functions monotonic---other than guaranteeing convexity in single-layer networks---or is this indeed mostly a "meme"?

Goodfellow tried Gaussian activation functions exp(-z^2) with both forms but I don't think he included it in the paper:

  • If z=wTx (a linear function), then the model was still highly vulnerable to adversarial examples.
  • If z=||w-x||2 (a quadratic template matching function), then the result depended on the depth of the model.
    • Shallow models worked OK (for their depth) and were noticeably resistant to adversarial examples.
    • Deep models were too difficult to train.

* See Goodfellow et al, 2015 and Hein et al, 2019

** Wu, Huaiqin (2009). "Global stability analysis of a general class of discontinuous neural networks with linear growth activation functions".

Update: Found some sinusoidal activation functions work, but still interested in RBF-like ones as well.

  • 1
    $\begingroup$ RBF networks do exist. apps.dtic.mil/dtic/tr/fulltext/u2/a196234.pdf I wonder if they're more challenging to train, because you have to select "good" centers for the RBF functions to make headway, and an exponentially decaying gradient implies that the vanishing gradient problem is severe. Just a guess, though. // Some of the universal approximation theorems for neural nets stipulate monotonic activation functions, so that could be one reason monotonicity persists. $\endgroup$ – Sycorax May 26 '20 at 16:37
  • $\begingroup$ @Sycorax, but Broomhead & Lowe's RBF networks are not trained by backprop. They are essentially just a linear network with a nonlinear transformation of the input (much as applying powers of your input variables). $\endgroup$ – seanv507 May 26 '20 at 16:47
  • $\begingroup$ @seanv507 right, what I'm guessing is that the reason that RBF networks aren't widely used today is that they're challenging to train with backprop. If you use a Broomhead & Lowe network, you're not really taking advantage of all the more recent innovations in NNs (deeper networks and so on) -- as you note. The citation is there because RBF networks do exist, they're just not widely used. $\endgroup$ – Sycorax May 26 '20 at 16:56
  • 1
    $\begingroup$ so I would say the reason Broomhead & Lowe networks are not widely used today is that they have basically been replaced by gaussian SVM, which essentially place a center at each data point ...(but again are not trained by backprop) $\endgroup$ – seanv507 May 26 '20 at 18:25
  • 1
    $\begingroup$ I already mentioned RBF networks in my original answer but I was hoping for MLPs with RBF activations, i.e. dot product w^T x instead of L2 norm ||x-w|| :) unless those two can be expressed as the same thing? $\endgroup$ – Christabella Irwanto May 26 '20 at 19:43

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.