Machine learning methods for multi-dimensional input and output I have a large dataset where my input is an $M$-dimensional tensor, and each input has a corresponding $N$-dimensional output. My goal is to train a method to learn outputs from the millions of inputs (i.e. tensors) in my database. Each of the elements composing the input and output tensors are simply floating point numbers, and this is essentially a regression task. In my specific case, $M$ and $N$ are $3$ and $2$, respectively (e.g. input is $20 \times 20 \times 20$ and output is $100 \times 100$). But generalized approaches are sought, if possible.
Based on past questions, there are suggestions to possibly convert the data into a single vector, although this appears to lose information about structural features (e.g. profiles). I'm also looking into convolutional neural networks, but treatment of multi-dimensional outputs appears missing. Overall, I am searching for methods and/or packages that can handle tensors of variable size as inputs and outputs (i.e. to perform a mapping in $M$-dimensions to $N$-dimensions).  
 A: From what it sounds to me you need to make a custom architecture for your task. Multi-dimensional outputs, however, should not bother you, as they are found in a multitude of tasks.
CNNs have been used for multi-dimensional outputs in many instances.
To name a couple:


*

*Probably the most common form of multi-dimensional input and output model is an Autoencoder. While this has little to do with your task, as its main goal is data compression, it is an architecture that takes an image (e.g. $224 \times 224 \times 3$) as an input and produces the same image as an output ($224 \times 224 \times 3$).

*Another task that requires both multi-dimensional input and output tesors is the task of image segmentation. Here the input, again, is an image (e.g. $224 \times 224 \times 3$), while the output is a number of segmentation masks ($224 \times 224 \times num\_classes)$. Some NN architectures that solve this task can be found here.
However, a CNN-based approach might under-perform if your input isn't image related, or doesn't have spatial features. 
Design tips:
If you decide to try it out, I'd like to give out a few pointers. Most architectures can be split into two parts: the first half of the model aims at compressing the input and extracting all possible information, while the second takes this compressed form and tries to recreate the desired output. 
For the first part, you can choose an architecture that has enough depth to sufficiently model the problem. Imagine you want to solve a simple binary classification task on your input data. What architecture would you choose? Drop the output layer and you have your first half. For the second part, try to gradually upscale this compressed form to yhe desired output. You can see how an architecture like U-Net handles this.
The output layer, in your case, needs to have a shape of $(100, 100)$ (+ the batch dimension). Also don't use any activation functions on this layer, because you are solving a regression problem.
