4
$\begingroup$

I'm curious if there are any neural network packages out there that easily allow one to construct feed forward neural networks with shared weights, but also allow for the training to be done in parallel. Torch7 allows for easy construction of shared weights, although the parallel training support is either not there (or not documented well enough to make it obvious that it is). If it interfaced to Python, that would be even better, but this is not a requirement.

$\endgroup$
3
$\begingroup$

It sounds like Pylearn2 may do what you want. It has two implementations for convolutional networks which requires some amount of weight sharing. Furthermore one of these implementations is optimized for use on a GPU via CUDA using Theano. See the documentation for more information. I believe the associated code is pylearn2.models.maxout.MaxoutConvC01B.

$\endgroup$
  • 1
    $\begingroup$ Thanks for the info DaemonMaker. Can you point me to the associated place in the documentation which specifies this? I couldn't find it at first pass, although I haven't fully delved into the full code yet. If it looks good I'll mark this as the answer. $\endgroup$ – gammapoint May 2 '14 at 14:40
  • $\begingroup$ Updated my answer. Feel free to let me know if you have additional questions. $\endgroup$ – DaemonMaker May 2 '14 at 21:06
5
$\begingroup$

I can't add a comment to @Emre's answer because I don't have enough points. You can train shared-weight networks in torch, be that using CUDA or not. The weight-sharing is supported for any tensor type. Training is done in parallel when you wrap the two shared modules in a nn.Parallel container

We use this in torch quite a lot, to build siamese networks.

$\endgroup$
  • $\begingroup$ Hi @nicko. Thanks for your answer. What torch package do you use exactly for the CUDA runs? And would this network setup work in parallel from your experience? I have a ParallelTable, although it's added to an nn.Sequential. Or do I need to modify things from as I have them? $\endgroup$ – gammapoint May 13 '14 at 22:43
  • $\begingroup$ Sorry, just saw this comment. I'll reply to you on the torch forums. short answer, you can use the "cunn" package to train your network, but i need a little more info on your network. $\endgroup$ – smhx May 14 '14 at 17:11
  • $\begingroup$ Hey @smhx, you seem very expert of siamese neural networks. Could you take a look to this question of mine (stackoverflow.com/questions/30581199/…) please? Thanks $\endgroup$ – DavideChicco.it Jun 1 '15 at 20:16
1
$\begingroup$

I don't have practical experience with torch7, but I think the idea is to do the parallelization on the GPU, cf. cunn and cutorch.

There's also https://github.com/clementfarabet/lua---parallel

$\endgroup$
  • $\begingroup$ Thanks Emre. What you say is also my understanding, although unfortunately I think that the cunn GPU implementations only cover a small subset of the full torch7 package, and I think the shared weight modules are not included in that. $\endgroup$ – gammapoint May 2 '14 at 14:38

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.