Why aren't neural networks used with RBF activation functions (or other non-monotonic ones)?

Question

In most work I've seen, MLPs (multilayer perceptron, the most typical feedforward neural network) and RBF (radial basis function) networks are compared as distinct models, where

• MLP neuron outputs $\sigma(\mathbf{w}^\top \mathbf{x})$.
  • $\sigma$ is a nonlinearity/activation function, e.g. ReLU, sigmoid, tanh
• RBF network neuron outputs $\phi_{k}\left(\left\|\mathbf{x}-\mathbf{c}_{k}\right\|_{2}\right)$
  • $\phi_k$ is also an activation function, but an RBF function
  • instead of dot product between inputs and weights, does euclidean distance
  • weights are interpreted as centers $\mathbf{c}_k$

I have 2 slightly-related questions:

1) Why aren't RBF activation functions $\phi(\cdot)$ ever used in MLPs, i.e.

$$\phi(\mathbf{w}^\top \mathbf{x})?$$

2) Or are RBF networks actually covered in the general MLP formulation $\sigma(\mathbf{w}^\top \mathbf{x})$ where $\sigma=\phi$, just by choosing different priors on w and b?

All the popular nonlinearities are monotonic (or if not, they are "almost-monotonic" e.g. Mish activation function, Google's Swish/SIL/SiLU, GELU). However, this answer by Toni Bellamo shows that non-monotonic activation functions work well too. Yoshuo Bengio in that same question also agrees. Furthermore, there is also a lot of research exposing fundamental flaws with the popular, monotonic ReLU-like activation functions*.

Is it so that, as these lecture notes put it,

the rarely questioned restriction to monotonic activation functions constitutes a surprisingly persistent meme that arose in the 1940s

Are there other reasons to keep functions monotonic---other than guaranteeing convexity in single-layer networks---or is this indeed mostly a "meme"?

Goodfellow tried Gaussian activation functions exp(-z^2) with both forms but I don't think he included it in the paper:

• If z=wTx (a linear function), then the model was still highly vulnerable to adversarial examples.
• If z=||w-x||2 (a quadratic template matching function), then the result depended on the depth of the model.
  • Shallow models worked OK (for their depth) and were noticeably resistant to adversarial examples.
  • Deep models were too difficult to train.

---

* See Goodfellow et al, 2015 and Hein et al, 2019

** Wu, Huaiqin (2009). "Global stability analysis of a general class of discontinuous neural networks with linear growth activation functions".

---

Update: Found some sinusoidal activation functions work, but still interested in RBF-like ones as well.

Answer 1

Seems that some people are a bit confused by your question because of its title; indeed, RBF neural networks exist, but they are a different architecture than the traditional multi-layer NNs. The body of your question is complete and formulated well (apart from some ambiguity because of the use of MLP), so to get to the answer let me ask a slightly different question: why are some activation functions more popular than others? Why are there only some activation functions in use? There is an infinite number of functions that are not used as activation functions in multi-layer NNs. One could think about some activation function that is a slight modification of the popular one, for example by squaring the argument, and ask "Why this kind of a function is not widely used as an activation function?"

In multi-layer NNs, each layer transforms the space in some (usually non-linear) way. When we use a non-monotonic activation function, we can imagine how a single layer of neurons works. But what would happen if we used another layer with a non-monotonic activation function transforming the outputs of the first layer? What if we added yet another such layer?

• As you can imagine, the transformation of the original space gets more sophisticated and twisted more quickly compared to if we used a monotonic activation function.
• The training process may also be more difficult because the error landscape is more complicated.
• Calculating the derivatives may be less efficient depending on the formula of the activation function.

So it is not like you cannot use a Gaussian as an activation function in a multi-layer NN. You can use it, and you can use other unpopular activation functions, and for a particular data and a particular NN topology they may even be more efficient and yield better predictions than those provided by the most popular activation functions. The question is: Why use this function and not the other? Why is this particular activation function beneficial? How is this function better from the other one that works well in a general case, hence it became so popular?

In a similar vein, you can use different activation functions (other than the most popular Gaussian) in the RBF NN architecture, and some of them will make more sense (and will perform better) than others.

This reminds me of another question that I want to bring up here just for illustration: how many layers in a NN you should use? We know that a single hidden layer is sufficient. So why people use more hidden layers? For efficiency, because of the lower total number of neurons needed when more layers are introduced? For better generalization? This demonstrates that your goal may be achieved in many ways, and it is good to first define some evaluation criteria to be able to measure the performance of different possible approaches and compare them.

Coming back to the original question "Why aren't [traditional] neural networks used with RBF activation functions (or other non-monotonic ones)?" – They are, but if one wants to use many such layers, given the potential problems I mentioned above, one should justify this decision by stating "This activation function is better than sigmoid or ReLU because..."