5
$\begingroup$

$\sin(x)$ seems to zero centered which is a desirable property for activation functions. Even the gradient won't vanish at any point. I am not sure if the oscillating nature of the function or its gradient can cause any issue during backpropagation.

$\endgroup$
8
$\begingroup$

Here's a paper dedicated to this very question:

Parascandolo and Virtanen (2016). Taming the waves: sine as activation function in deep neural networks.

Some key points from the paper:

  • Sinusoidal activation functions have been largely ignored, and are considered difficult to train.

  • They review past work that has used sinusoidal activation functions. Most of this is earlier work (before the modern 'deep learning' boom). But, there are a couple more recent papers.

  • The periodic nature of sinusoidal activation functions can give rise to a 'rippling' cost function with bad local minima, which may make training difficult.

  • The problem may not be so bad when the data is dominated by low-frequency components (which is expected for many real-world datasets). Learning is easier in this regime, but is sensitive to how network parameters are initialized.

  • They show that networks with sinusoidal activation functions can perform reasonably well on a couple real-world datasets. But, after training, the networks don't really use the periodic nature of the activation function. Rather, they only use the central region near zero, where the sinusoid is nearly equivalent to the more traditional $\tanh$ activation function.

  • They trained recurrent networks on a synthetic task where periodic structure is expected to be helpful. Networks learn faster and are more accurate using $\sin$ compared to $\tanh$ activation functions. But, the difference is bigger for vanilla RNNs than LSTMs.

Here's another relevant paper:

Ramachandran, Zoph, Le (2017). Searching for Activation Functions.

They performed a large-scale, automatic search over activation functions to find new variants that perform well. Some of the activation functions they discovered use sinusoidal components (but they're not pure sinusoids--they also tend to have a monotonic component). The paper doesn't discuss these variants much, except to say that they're an interesting future research direction.

$\endgroup$
  • $\begingroup$ I wonder how much of the utility of sinusoidal activation functions is related to so-called "random kitchen sinks" (people.eecs.berkeley.edu/~brecht/kitchensinks.html) and whether or not a fixed sinusoidal basis expansion followed by a neural network would be a strong learner. Do you know of any literature that addresses this? $\endgroup$ – Sycorax Apr 12 at 14:42
3
$\begingroup$

It surely "can" be used. The fact that it is not being used1, however, suggests that it is not very much practical. The gradient of $\sin$ is actually zero at $\frac \pi 2+k\pi$ for any integer $k$. I think the main problem with using $\sin$ activation is that it introduces infinitely many symmetries, which may make the learning even harder than it is.

However, the right architecture always depends on the specific problem and data representation that you are dealing with. I am sure there are scenarios where tailoring $\sin$ into the network may not only make sense, but also improve the performance.


1 That is, not being commonly used. To the extent that I have not seen it ever being used (though my experience is limited to computer vision problems). It certainly is not one of the generally applicable nonlinearities such as ReLU or sigmoid.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.