Can $\sin(x)$ be used as activation in deep learning? $\sin(x)$ seems to zero centered which is a desirable property for activation functions. Even the gradient won't vanish at any point. I am not sure if the oscillating nature of the function or its gradient can cause any issue during backpropagation.
 A: Here's a paper dedicated to this very question:

Parascandolo and Virtanen (2016). Taming the waves: sine as activation function in deep neural networks.

Some key points from the paper:


*

*Sinusoidal activation functions have been largely ignored, and are considered difficult to train.

*They review past work that has used sinusoidal activation functions. Most of this is earlier work (before the modern 'deep learning' boom). But, there are a couple more recent papers.

*The periodic nature of sinusoidal activation functions can give rise to a 'rippling' cost function with bad local minima, which may make training difficult.

*The problem may not be so bad when the data is dominated by low-frequency components (which is expected for many real-world datasets). Learning is easier in this regime, but is sensitive to how network parameters are initialized.

*They show that networks with sinusoidal activation functions can perform reasonably well on a couple real-world datasets. But, after training, the networks don't really use the periodic nature of the activation function. Rather, they only use the central region near zero, where the sinusoid is nearly equivalent to the more traditional $\tanh$ activation function.

*They trained recurrent networks on a synthetic task where periodic structure is expected to be helpful. Networks learn faster and are more accurate using $\sin$ compared to $\tanh$ activation functions. But, the difference is bigger for vanilla RNNs than LSTMs.
Here's another relevant paper:

Ramachandran, Zoph, Le (2017). Searching for Activation Functions.

They performed a large-scale, automatic search over activation functions to find new variants that perform well. Some of the activation functions they discovered use sinusoidal components (but they're not pure sinusoids--they also tend to have a monotonic component). The paper doesn't discuss these variants much, except to say that they're an interesting future research direction.
A: It surely "can" be used. The fact that it is not being used1, however, suggests that it is not very much practical. The gradient of $\sin$ is actually zero at $\frac \pi 2+k\pi$ for any integer $k$. I think the main problem with using $\sin$ activation is that it introduces infinitely many symmetries, which may make the learning even harder than it is.
However, the right architecture always depends on the specific problem and data representation that you are dealing with. I am sure there are scenarios where tailoring $\sin$ into the network may not only make sense, but also improve the performance.

1 That is, not being commonly used. To the extent that I have not seen it ever being used (though my experience is limited to computer vision problems). It certainly is not one of the generally applicable nonlinearities such as ReLU or sigmoid.
A: As of June 2020, work from Stanford has demonstrated that your intuition was right and that $sin(x)$ can be indeed used for a variety of tasks effectively:

*

*https://vsitzmann.github.io/siren/

*https://arxiv.org/pdf/2006.09661.pdf
A: This was a part of my master thesis. I have researched and mentioned previous publications on the use of sine function as activation and as the basis function. In addition, I have designed several different versions of the sine function and used them as the basis function in the artificial neural networks as a replacement for the linear function. It was last year. You can find the paper that represents the "use of sine" part of my thesis in the link: CSIMQ
I mentioned in the paper, but you can also search for Fourier Neural Networks for this subject. The artificial neural networks that use sine function as a basis are named like that mostly.
In addition, I have shared several articles in Medium regarding this subject that can help you.
Creating Alternative Truths with Sine Activation Function in Neural Networks
Sinusoidal Neural Networks for Digit Classification
Neural Networks with Sine Basis Function
