# Machine learning methods for multi-dimensional input and output

I have a large dataset where my input is an $M$-dimensional tensor, and each input has a corresponding $N$-dimensional output. My goal is to train a method to learn outputs from the millions of inputs (i.e. tensors) in my database. Each of the elements composing the input and output tensors are simply floating point numbers, and this is essentially a regression task. In my specific case, $M$ and $N$ are $3$ and $2$, respectively (e.g. input is $20 \times 20 \times 20$ and output is $100 \times 100$). But generalized approaches are sought, if possible.

Based on past questions, there are suggestions to possibly convert the data into a single vector, although this appears to lose information about structural features (e.g. profiles). I'm also looking into convolutional neural networks, but treatment of multi-dimensional outputs appears missing. Overall, I am searching for methods and/or packages that can handle tensors of variable size as inputs and outputs (i.e. to perform a mapping in $M$-dimensions to $N$-dimensions).

• "multi-dimensional outputs appears missing." Why do you say that? Neural Networks can easily have multiple output neurons instead of one output neuron. Aug 9 '18 at 0:21
• @Alex I may simply be mistaken, but after searching for examples, I could not find instances of neural nets taking tensors as both inputs and outputs. If you can provide particular code examples, that would be greatly appreciated. Also, it's perhaps worth adding that I am interested in learning of methods in addition to neural networks to tackle this problem. But any solution is still a solution. Aug 9 '18 at 0:37

From what it sounds to me you need to make a custom architecture for your task. Multi-dimensional outputs, however, should not bother you, as they are found in a multitude of tasks.

CNNs have been used for multi-dimensional outputs in many instances.
To name a couple:

• Probably the most common form of multi-dimensional input and output model is an Autoencoder. While this has little to do with your task, as its main goal is data compression, it is an architecture that takes an image (e.g. $224 \times 224 \times 3$) as an input and produces the same image as an output ($224 \times 224 \times 3$).

• Another task that requires both multi-dimensional input and output tesors is the task of image segmentation. Here the input, again, is an image (e.g. $224 \times 224 \times 3$), while the output is a number of segmentation masks ($224 \times 224 \times num\_classes)$. Some NN architectures that solve this task can be found here.

However, a CNN-based approach might under-perform if your input isn't image related, or doesn't have spatial features.

### Design tips:

If you decide to try it out, I'd like to give out a few pointers. Most architectures can be split into two parts: the first half of the model aims at compressing the input and extracting all possible information, while the second takes this compressed form and tries to recreate the desired output.

For the first part, you can choose an architecture that has enough depth to sufficiently model the problem. Imagine you want to solve a simple binary classification task on your input data. What architecture would you choose? Drop the output layer and you have your first half. For the second part, try to gradually upscale this compressed form to yhe desired output. You can see how an architecture like U-Net handles this.

The output layer, in your case, needs to have a shape of $(100, 100)$ (+ the batch dimension). Also don't use any activation functions on this layer, because you are solving a regression problem.

• This was incredibly helpful and I'm going to look into develop a custom architecture. Would you happen to have any suggestions for methods besides CNNs as well? Aug 10 '18 at 18:45
• Unfortunately not, this works well with the "deep" nature of CNNs, where you have lots of layers, stacked one after another. If you use let's say fully connected layers, the model would be untrainable due to the large number of parameters (high memory requirements, slow computation time, etc.)... Aug 11 '18 at 1:38
• This is something I will investigate as I move forward, but is there any limit I should keep in mind in terms of layers for the CNN? Aug 11 '18 at 5:13
• You could go for a popular U-Net like style where you have blocks of 2 convolution layers followed by a pooling layer. These three layers result in lowering the size of the input dimensions by half. Because you start from 20, you don't want to go too low; I'd try two such blocks, where the first would go from 20 to 10 and the second from 10 to 5. The number of filters (i.e. the third dimension), however should be doubled in each such block (20 to 40 in the first and 40 to 80 in the second). Also something I just thought of, if you have volumetric data you can look into a 3D-UNet architecture. Aug 11 '18 at 11:43