How can I build a 6D Convolution Layer in a CNN, using R or Python? As stated in the title, I would like to build an N-dimensional Convolutional Layer as part of a Convolution Neural Network, without doing dimensionality reduction on my data; because I have multiple spatial features. I am a bit stumped as to what the best package to build one would be, or where to start. It looks like both Pytorch and Tensorflow lack the ability to create this type of high dimensional convolution layer.
E.g. Imagine I have 6 spatial dimensions, and I want to input them to a 6-dimensional CNN layer.
To be clear, I'm dealing with high-dimensional but manageable counts, and I am okay with relatively long training times. This is mostly an exercise in curiosity, at the moment.
EDIT: To be more precise, what I have is 3D vectors in a relatively sparse 6D space with short dimensions. I definitely want to use a true 6D convolution in the first layer. That is the specific aim of this exercise. I am just looking for the mechanics of how to achieve this. It seems Nvidia/MinkowskiEngine on GitHub might address my dilemma.
https://github.com/NVIDIA/MinkowskiEngine
 A: Time series of a coordinate in 6D space
It might be that one has 6+1D coordinates for time and space but one only wants to convolve along the time direction.
For instance, if derivatives are important, then your code will learn to compute convolutions like
$$\begin{array}{llllllllll}
x^\prime_t& \approx &  +1 x_t &-1 x_{t-1}\\
x^{\prime\prime}_t 
&\approx& +1 x_t &- 2 x_{t-1} &+1x_{t-2}\end{array}$$ and you could get more precise coefficients that resemble some Savitky-Golay filter.
You could also perform a multidimensional convolution, those type of filters have multidimensional equivalents, but if you do not have a 6D image (all points in the 6D space have a value), and instead only a (time-)curve of coordinates inside that space, then using 1D convolutions seems better (much less computations).
The case of a multidimensional kernel, but only integration steps in a few directions.
In the comments a case is described of a 6D discrete space where the some axes have only Boolean or few values.
In that case one can reshape the space and perform the convolution with a stride.
Example in 2x6 cases convolved with a 2x2 kernel
$$\text{input$_{2\times6}$} = \begin{bmatrix}{}
x_{11}&x_{12}&x_{13}&x_{14}&x_{15}&x_{16}\\
x_{21}&x_{22}&x_{23}&x_{24}&x_{25}&x_{16}\\
\end{bmatrix}$$
If you properly reshape this then you get
$$\text{reshaped input$_{1\times12}$} = \vphantom{\begin{bmatrix}\rlap{{\overbrace{\phantom{x_{11}\,x_{21}\,x_{12}\,x_{22}}}^{\substack{\text{first step operates}\\\text{on these four}}}}}x_{11}\, x_{21}{\underbrace{x_{12}\,x_{22}\,x_{13}\,x_{23}}_{\substack{\text{second step operates}\\\text{on these four}}}}\,x_{14}\,x_{24}\,x_{15}\,x_{25}\,x_{16}\,x_{26}\end{bmatrix}}
\begin{bmatrix}\rlap{\smash{\overbrace{\phantom{x_{11}\,x_{21}\,x_{12}\,x_{22}}}^{\substack{\text{first step operates}\\\text{on these four}}}}}x_{11}\, x_{21}\smash{\underbrace{x_{12}\,x_{22}\,x_{13}\,x_{23}}_{\substack{\text{second step operates}\\\text{on these four}}}}\,x_{14}\,x_{24}\,x_{15}\,x_{25}\,x_{16}\,x_{26}\end{bmatrix}$$
And your convolution will have a kernel size of 1x4 and a stride of 2. In the above expression I have shown this for the first two steps of the convolution, and eventually the output will be of size $6-1 = 5$.
Alternatively, you convolve over your desired subset of $n$ dimensions and treat the other $6-n$ dimensions as defining different channels (such that the kernel differ per channel), then afterwards you apply a layer that adds the channels together (effectively ending up with a convolution along $n$ dimensions with a 6D kernel).
More generally if you have a space of shape/size $x_1 \times x_2 \times x_3 \times x_4 \times x_5 \times x_6$ and you convolve with a kernel of shape/size $k_1 \times k_2 \times k_3 \times k_4 \times k_5 \times k_6$ then you end up with dimensions of size $y_i = x_i - (k_i-1)$. If also some of the kernel dimensions have the same length as the input shape the (ie if $x_i = k_i$), then you will end up with output shape that has dimensions of length one $y_i = 1$. If you have an output shape with more than 3 of the 6 dimensions of length one, then you can use one of the lower dimensional convolution layers.
In the example, the $2 \times 6$ shape is effectively a 1D convolution with an output shape of $1 \times 5$.
