# Best way to represent 3D data for Neural Networks

I want to train a generative model over a dataset where each example is a $X = (N,3)$ matrix representing $N$ points in $\mathbb{R}^3$.

The local structure (i.e. the correlations between neighboring points, as measured by the Euclidean distance in $\mathbb{R}^3$) is very important.

For now I am representing this data directly as a $(n_{batch},3,N)$ tensor, so the spatial dimensions are treated as channels in the convolutional layers of the model.

I am wondering if this is the best representation, since even for $N=200$ a convolutional network would need to be very deep to have a receptive field of this size as to be able to catch correlations between all points. (Far away points in the $N$ dimension car be spatially close).

I thought about other ways to represent this data :

The first is to take the inner product matrix or Gram matrix defined as $G_{ij} = X_i . X_j$ which results in a $(N,N)$ matrix. But these matrices are positive semi-definite (SPD), and building a Neural Network that preserves this property is very expensive.

The second is to take the outer product matrix defined as $T_i = X_iX_i^T$ which results in a $(N,3,3)$ matrix. I am not sure about the properties and the advantages of this representation.

Question: Are there other tensor representations of $3D$ points that would be better suited to the extraction of local correlations ?

Note : An important feature of such representations is that the original matrix should be easily recovered.

• I am not sure about what is the representation you suggest : how could I represent $N$ points in $\mathbb{R}^3$ as an $(N,N,N)$ tensor ? Are you suggesting to implement $3D$ convolutions (so operate on a 5D tensor, considering a dummy dimension for the channels) ? Still, your suggestions are interesting, thank you. Commented Mar 20, 2018 at 15:11