Is there a 3D neural network and how to code it in R?

Question

I have coded some NN using neuralnet and caret packages. But these neural networks are of 2-Dimension (I dont know even this term exists)

But I was curious whether "3D neural network" exists. While searching I came across this website and found this neural network.

I have a good understanding of feed forward and back propagation concepts in neural network. If a 3d neural network exist how does it work?Is there any package in R which can be used to create models(and predictions) for this type of neural network? If not can you suggest any other platform?

Answer 1

A '3d network' might commonly be described as a network with 2d layers. It's not fundamentally different from any other network because the principles of activation are the same. The activation of each unit is a linear combination of its inputs, passed through a (typically nonlinear) activation function. In one sense, the dimensionality is just a property of the drawing. You could draw the units anywhere you wanted (using however many dimensions you wanted) and the function would be the same. Instead, it's the connectivity that matters.

There are a couple reasons to 'organize' networks layers into well defined shapes. One is for convenience in thinking about things (e.g. in the case of processing 2d inputs like images). This is particularly the case when connectivity is constrained. For example, in a convolutional network that processes images, each unit receives connections from a local 'patch' of units in the previous layer. Thinking about the layers as 2d makes intuitive sense here because it lets us talk about things like 'local patches'. But, as before, you could completely scramble the 'locations' and the function would be the same as long as connectivity is the same.

In convolutional networks (e.g. for image processing), there's an additional benefit to representing the layers in 2d. Because of the way these networks constrain the weights/connectivity, representing the layers in 2d makes it possible to use the 2d convolution operation when computing the activations of all units in a layer. Although it doesn't change the fundamental function of the network, it's a very computationally efficient way of implementing things. This is just one example, and there could be other cases where representing units on a grid with some dimensionality makes it possible to play computational tricks that speed things up (e.g. in the blog post you linked).

Actually, the network in the blog post is a different beast because it's a spiking network that tries to emulate biological neurons slightly more closely than a standard artificial neural net. But, that's a whole different issue.

Answer 2

For artificial neural networks (the kind employed in machine learning) there is no "dimensionality". As @user20160 notes, convolution nets are often presented in 2D to help us understand the operations of the network, but there is no position in space for any of the units, just connections to different parts of an image. In the website you link to, the neural network has a connectivity rule that is defined by unit-to-unit hop distances, meaning that there is an implicit 3D spatial location for each unit. To answer your question: I don't think there are any packages in R for this type of architecture (I haven't done an exhaustive search though). But, the NEURON or Brian simulation environments could potentially let you do it with simple integrate-and-fire units. As well, implementing it in R or python wouldn't be that hard - just define a point in space for each neuron and set a connectivity rule based on distance.

I would note: whether such a design is useful is an open question. We see some patterns of connectivity the depend on distance in the real brain (see e.g. this paper), so there may be a good reason to do it. As far as I know, though, no one has ever actually demonstrated a good reason from a machine learning perspective.