Neural networks with complex weights

Question

I am currently wishing to give a neural network starting weights with complex values (because of the nature of the specific task I am working with). I was trying to use the standard neural net libraries from scikit learn and the optimization functions it uses do not work with complex numbers. Thus I am confronting the following problem:

Optimize a function numerically $f(z)$ where $z \in \mathbb{C} $ .

I am doing this via numerical packages (specifically scipy in python) and I have noticed that all the optimization methods in this package are tailored to only functions of domain in $\mathbb{R}$. I Played around for a bit with the complex optimization problem and at first sight it just seems like numerically optimizing a function of two variables since $z = (x,y) \equiv z = x + iy$ .

I am aware that the definition of an analytic function is more rigorous than that of a real differentiable function in the sense that it should be differential from all directions $\Delta z = (\Delta x, \Delta y)$ so things become a bit more complicated. Given the situation above how can one approach the task of numerical optimization on a function of complex domain? Are there any standard routines for this ?

Are there any works/ standard methods for neural networks and gradient descent with complex weights ?

Answer 1

Gradient Descent will work in your case. You can use Theano. It supports differentiation with respect to complex variables. You can check this link if you need more information. One important note is that your error function needs to return a scalar.

I'm not quit sure that other algorithms like Conjugate gradient or quasi-Newton will work fine for complex numbers. If you want to implement other learning algorithms, you will need to verify their proves to make sure that they are possible as well.

Answer 2

The differences between complex-valued and real-valued networks have been thoroughly explained in an ESANN 2011 paper by Zimmermann, Minin and Kusherbaeva (Comparison of the Complex Valued and Real Valued Neural Networks Trained with Gradient Descent and Random Search Algorithms) which reading I recommend.

Main points :

• The feed-forward function may have singularity points ; you can avoid that by using a sigmoid complex function that has a singularity at infinite values only, and use a bounded range for your inputs to stay safe ;

• The back-propagation algorithm can be recovered by basing the weight adaptation procedure on the Taylor expansion for the error $E(W+\Delta W)= E(W) + G^T\Delta W + 1/2 \Delta W^T G \Delta W $ then the rule for weight adaptation can be written as $\Delta w = -\eta . \delta E/\delta W $ with $\eta$ as learning rate.

• The sensitivity to initial conditions led the authors of the article to recommend the use of RSA (Random Search Algorithm) rather than direct random initialization of the weights.

• Once RSA is used the complex-valued neural network will always converge, and is reported to be on par with real-valued network (slightly better but not significantly in the experiments).

In conclusion : you can use complex weights in neural networks if your domain requires it. However you should devote some time to build a sound specific library, as there are subtle pitfalls.