# Best way to represent 3D data for Neural Networks

I want to train a generative model over a dataset where each example is a $X = (N,3)$ matrix representing $N$ points in $\mathbb{R}^3$.

The local structure (i.e. the correlations between neighboring points, as measured by the Euclidean distance in $\mathbb{R}^3$) is very important.

For now I am representing this data directly as a $(n_{batch},3,N)$ tensor, so the spatial dimensions are treated as channels in the convolutional layers of the model.

I am wondering if this is the best representation, since even for $N=200$ a convolutional network would need to be very deep to have a receptive field of this size as to be able to catch correlations between all points. (Far away points in the $N$ dimension car be spatially close).

I thought about other ways to represent this data :

The first is to take the inner product matrix or Gram matrix defined as $G_{ij} = X_i . X_j$ which results in a $(N,N)$ matrix. But these matrices are positive semi-definite (SPD), and building a Neural Network that preserves this property is very expensive.

The second is to take the outer product matrix defined as $T_i = X_iX_i^T$ which results in a $(N,3,3)$ matrix. I am not sure about the properties and the advantages of this representation.

Question: Are there other tensor representations of $3D$ points that would be better suited to the extraction of local correlations ?

Note : An important feature of such representations is that the original matrix should be easily recovered.

## 1 Answer

3D images are unfortunately rather annoying in their complexity. Your best bet is to review the literature, especially medical literature as most human scans are three dimensional. A great example is the Kaggle Lung competition. It sounds like your data is grayscale so that's a good start. There are plenty of tricks to making this scalable:

1) Make sure your batchsize is well-regulated. 3d images and their computations are huge, which will blow up most GPUs with OOM errors.

2) Scale down the initial image as much a possible. Currently around 128x128x128 is about as large as you can go. If your image must be bigger, you can split it into such regions (possibly with overlap) and run your model on each of them.

3) Further simplification includes assuming that each volumetric direction has the same structure, so you can use the same convolutional weights in the x-y plane as also in the y-z plane.

4) Since you're training a generative model, you probably need to sacrifice even more on your input size, maybe down to 32x32x32.

• I am not sure about what is the representation you suggest : how could I represent $N$ points in $\mathbb{R}^3$ as an $(N,N,N)$ tensor ? Are you suggesting to implement $3D$ convolutions (so operate on a 5D tensor, considering a dummy dimension for the channels) ? Still, your suggestions are interesting, thank you. – Toool Mar 20 '18 at 15:11