As stated in the title, I would like to build an N-dimensional Convolutional Layer as part of a Convolution Neural Network, without doing dimensionality reduction on my data; because I have multiple spatial features. I am a bit stumped as to what the best package to build one would be, or where to start. It looks like both Pytorch and Tensorflow lack the ability to create this type of high dimensional convolution layer.

E.g. Imagine I have 6 spatial dimensions, and I want to input them to a 6-dimensional CNN layer.

To be clear, I'm dealing with high-dimensional but manageable counts, and I am okay with relatively long training times. This is mostly an exercise in curiosity, at the moment.

EDIT: To be more precise, what I have is 3D vectors in a relatively sparse 6D space with short dimensions. I definitely want to use a true 6D convolution in the first layer. That is the specific aim of this exercise. I am just looking for the mechanics of how to achieve this. It seems Nvidia/MinkowskiEngine on GitHub might address my dilemma. https://github.com/NVIDIA/MinkowskiEngine

  • 2
    $\begingroup$ Why would you want to convolve along a translation-rotation space? When people say N-dimensional convolution, they mean that the convolutional filter is N-dimensional. How is your data organized? Also, be sure this isn't a XY problem $\endgroup$
    – Firebug
    Feb 8, 2023 at 8:25
  • $\begingroup$ @Firebug I am looking to predict the chaotic motion of bodies in a 3D space using both translation and rotation vectors. Think bouncy balls. Rotation vectors are grouped into a relatively small number of discrete buckets, just like translation vectors. As I said, this is a curiosity. I want to see how such a model might compare with predicting the motion of bouncing balls, vs using a 3D CNN with translation vectors only. $\endgroup$
    – David
    Feb 8, 2023 at 8:33
  • 2
    $\begingroup$ So your convolution is actually along time, it's a 1D convolution with 6 input channels $\endgroup$
    – Firebug
    Feb 8, 2023 at 8:51
  • 1
    $\begingroup$ Could you make an example of the convolution that you want to perform by writing it down as a mathematical formula. You have a 6D image that you want to integrate over? $\endgroup$ Feb 8, 2023 at 9:08
  • 1
    $\begingroup$ You don't have to describe your data, but it would be much easier if you described the input "tensor" (aka multidimensional array), in terms of spacetime-like dimensions and non-spacetime-like dimensions. Convolutions are usually performed along spacetime-like dimensions (e.g. Euclidean coordinates, time) over the rest of the variables. Then describe what the output should look like. $\endgroup$
    – Firebug
    Feb 9, 2023 at 8:39

1 Answer 1


Time series of a coordinate in 6D space

It might be that one has 6+1D coordinates for time and space but one only wants to convolve along the time direction.

For instance, if derivatives are important, then your code will learn to compute convolutions like $$\begin{array}{llllllllll} x^\prime_t& \approx & +1 x_t &-1 x_{t-1}\\ x^{\prime\prime}_t &\approx& +1 x_t &- 2 x_{t-1} &+1x_{t-2}\end{array}$$ and you could get more precise coefficients that resemble some Savitky-Golay filter.

You could also perform a multidimensional convolution, those type of filters have multidimensional equivalents, but if you do not have a 6D image (all points in the 6D space have a value), and instead only a (time-)curve of coordinates inside that space, then using 1D convolutions seems better (much less computations).

The case of a multidimensional kernel, but only integration steps in a few directions.

In the comments a case is described of a 6D discrete space where the some axes have only Boolean or few values.

In that case one can reshape the space and perform the convolution with a stride.

Example in 2x6 cases convolved with a 2x2 kernel

$$\text{input$_{2\times6}$} = \begin{bmatrix}{} x_{11}&x_{12}&x_{13}&x_{14}&x_{15}&x_{16}\\ x_{21}&x_{22}&x_{23}&x_{24}&x_{25}&x_{16}\\ \end{bmatrix}$$

If you properly reshape this then you get

$$\text{reshaped input$_{1\times12}$} = \vphantom{\begin{bmatrix}\rlap{{\overbrace{\phantom{x_{11}\,x_{21}\,x_{12}\,x_{22}}}^{\substack{\text{first step operates}\\\text{on these four}}}}}x_{11}\, x_{21}{\underbrace{x_{12}\,x_{22}\,x_{13}\,x_{23}}_{\substack{\text{second step operates}\\\text{on these four}}}}\,x_{14}\,x_{24}\,x_{15}\,x_{25}\,x_{16}\,x_{26}\end{bmatrix}} \begin{bmatrix}\rlap{\smash{\overbrace{\phantom{x_{11}\,x_{21}\,x_{12}\,x_{22}}}^{\substack{\text{first step operates}\\\text{on these four}}}}}x_{11}\, x_{21}\smash{\underbrace{x_{12}\,x_{22}\,x_{13}\,x_{23}}_{\substack{\text{second step operates}\\\text{on these four}}}}\,x_{14}\,x_{24}\,x_{15}\,x_{25}\,x_{16}\,x_{26}\end{bmatrix}$$

And your convolution will have a kernel size of 1x4 and a stride of 2. In the above expression I have shown this for the first two steps of the convolution, and eventually the output will be of size $6-1 = 5$.

Alternatively, you convolve over your desired subset of $n$ dimensions and treat the other $6-n$ dimensions as defining different channels (such that the kernel differ per channel), then afterwards you apply a layer that adds the channels together (effectively ending up with a convolution along $n$ dimensions with a 6D kernel).

More generally if you have a space of shape/size $x_1 \times x_2 \times x_3 \times x_4 \times x_5 \times x_6$ and you convolve with a kernel of shape/size $k_1 \times k_2 \times k_3 \times k_4 \times k_5 \times k_6$ then you end up with dimensions of size $y_i = x_i - (k_i-1)$. If also some of the kernel dimensions have the same length as the input shape the (ie if $x_i = k_i$), then you will end up with output shape that has dimensions of length one $y_i = 1$. If you have an output shape with more than 3 of the 6 dimensions of length one, then you can use one of the lower dimensional convolution layers.

In the example, the $2 \times 6$ shape is effectively a 1D convolution with an output shape of $1 \times 5$.

  • $\begingroup$ A 2D kernel will absolutely lose spatial information. If I line those long vectors up tail on tail, the relevant spatial information I am trying to capture will be lost (although it is still an open question as to how helpful the spatial information would be to the model - that remains to be tested). I appreciate your follow-ups, but as I stated earlier I'm really looking for how to go about creating a 6D kernel which maps the 6D data to a 6D layer, then I'll process subsequent layers down into fewer dimensions. It is this step I am a bit stuck on, and seeking assistance for. $\endgroup$
    – David
    Feb 9, 2023 at 21:49
  • $\begingroup$ @David the 2D kernel is just an example. You will do it of course in 6D. With the example I show a convolution in a 2D space that occurs only along a single direction. This is what you indicated is also your situation. You have a 6D space but the convolution only happens along a few dimensions. This means that the output is not 6D.... $\endgroup$ Feb 9, 2023 at 23:04
  • $\begingroup$ ...The coordinates have for several dimensions (certainly the dichotomic ones) only one value (and effectively the dimension doesn't play a role) because the kernel takes up the entire space along certain dimensions. In the example we convolve a 2D space of size $k \times l$ with a kernel of size $2 \times 2$, and we end up with an output size of $k-1 \times l-1$. More specifically the input is $2 \times 6$ and we end with output of size $1 \times 5$, reducing the dimension effectively by one. $\endgroup$ Feb 9, 2023 at 23:10
  • $\begingroup$ "A 2D kernel will absolutely lose spatial information." I am not saying that you should use a 2D kernel. I use a 2D kernel as example to demonstrate the lowering of the dimensions to perform a convolution with a 2D kernel in a 1D convolution. In your case the situation will be with a 6D kernel. The idea is that the dimension of the kernel is not limited to 2 or 3 when you use the 2d or 3d convolution layers. It can be as many dimensions as you want, it is however the directions of the convolution that are limited. $\endgroup$ Feb 9, 2023 at 23:15

Not the answer you're looking for? Browse other questions tagged or ask your own question.