# Neural Net for multivariate regression

I need to build a model (M) that converts a 10 dimensional space of inputs (A) into a 20 dimensional space of outputs B.

Both the inputs and outputs are analog, so this is not a classification problem but rather a regression one.

A * M = B

In order to allow some bias I included a last row of 1s in A:

$A = \begin{bmatrix} A & 1 \end{bmatrix}$

My first approach was to just to left multiply by the pseudoInverse of A to obtain M:

$M=A^{-1}B$

However this is not getting good results.

Analyzing what I did, I think it's equivalent to using a one layer neural network with bias but without activation function and without regularization.

Now, to improve my model I want to make it more complex using neural networks, but some questions arise.

I find a lot of examples in internet using neural networks for classification, where each output neuron represents a class and it's output is associated to the probability of the input belonging to that class.

However my problem has nothing to do with classification (think about it as an operator that takes some location in a 10 dimensional space and "moves" it to a new location in a higher dimensional space).

Is there some specific neural network topology (type of activation function, regularization, training method) that is specific for this problem? How would you approach such a problem?

Thanks a lot.

Before giving up on linear models, you could also try regularized linear models. For example, you can penalize the $\ell_2$ norm of the weights (ridge regression), which expresses a preference for smaller weights. You can also penalize the $\ell_1$ norm (lasso), which induces sparse solutions (you can think of this as a kind of automatic feature selection). There's also the elastic net, which is a combination of $\ell_1$ and $\ell_2$ penalties. These techniques are very popular, and can improve generalization performance when appropriately matched to the problem. They can also make it possible to solve problems where the number of input variables exceeds the number of data points. You'll often see these techniques discussed in the context of regression problems with a single scalar output. But, you can also apply them in the case of vector-valued outputs. If searching for sparse solutions, you'd have to decide whether or not the weights for all outputs should share the same sparsity structure (i.e. should all columns of $M$ have zeros in the same rows as each other?).