# Neural network for multiple output regression

I have a dataset containing 34 input columns and 8 output columns. One way to solve the problem is to take the 34 inputs and build individual regression model for each output column. I am wondering if this problem can be solved using just one model particularly using Neural Network.

I have used a multilayer perceptron, but that needs multiple models just like linear regression. Can sequence to sequence1 learning be a viable option? I tried using TensorFlow it does not appears to be able handle float values.

Any suggestion to tackle this problem by using just one unified model specially using neural network will be appreciated.

1. Ilya Sutskever, Oriol Vinyals, & Quoc V. Le (2014). Sequence to Sequence Learning with Neural Networks. Advances in Neural Information Processing Systems, 27. (pdf)
• What is the problem? Feb 11, 2017 at 0:37
• (To any potential close voters: This is not a programming question. It would be off topic on Stack Overflow.) Feb 11, 2017 at 0:38
• Can you clarify the "sequence to sequence" option here? I don't know if it will be familiar to people who work w/ NNs, but it isn't to me. Feb 11, 2017 at 0:42
• Sequence to Sequence modelling using RNN. papers.nips.cc/paper/… Feb 11, 2017 at 1:03
• You can try MultiOutputRegressor(). This works for this type of problems Jun 12, 2019 at 10:40

A neural net with multiple outcomes takes the form $$\mathbf{Y} = \mathbf{\gamma} + \mathbf{V}_1\Gamma_1 + \epsilon\\ \mathbf{V}_1 = a\left(\gamma_2 +\mathbf{V}_2\Gamma_2\right)\\ \mathbf{V}_2 = a\left(\gamma_3 +\mathbf{V}_3\Gamma_3\right)\\ \vdots \\ \mathbf{V}_{L-1} = a\left(\gamma_L+ \mathbf{X}\Gamma_L\right)\\$$ If your outcome has the dimension $N\times 8$, then $[\gamma_1, \Gamma_1]$ will have the dimension $(p_{V1}+1) \times 8$.

Which is to say that you'd be assuming that each outcome shares ALL of the parameters in the hidden layers, and only has different parameters for taking the uppermost derived variable and relating it to the outcome.

Is this a realistic assumption for your context?

At first I thought generic_user's comment was a show-stopper, but I just realized it isn't:

If I train d different networks on d different outputs, then each one will be fit to that dimension with no regard for the others.

But if I train one network with d outputs and use all outputs for backpropagation, then each weight in every layer in the network will be adjusted so that all d outputs are more accurate.

That is: Each network parameter will be adjusted by a sum of gradients (how each output varies with a "wiggle" in this parameter) such that adjusting it in the chosen up or down direction causes more accurate output overall--even if adjusting the weight that way causes some dimensions of the output to be less accurate.

So, yes, the thing that ultimately distinguishes each output is just a linear equation encoded in the last layer, but in training one multi-output network every layer will become better at presenting that last layer with something that allows it to do its job better. And so it follows that the relationships between outputs will be accounted for by this architecture.

You might be able to do better than a fully-connected net by making your architecture reflect any known relationships in the output, just as deep networks do better than shallow ones by exploiting "compositional" relationships between inputs.

• A multivariate regression assumes there is some (unknown) covariance matrix that relates that target variables to each other. Are you suggesting that just having one dense layer is enough to capture this covariance without the need for explicitly including hidden layer(s)? Nov 20, 2019 at 22:36
• I'm suggesting that hidden layers must capture some of that covariance. One dense output layer alone is just a bunch of parallel linear-regressors-through-activation-functions. No relationship between outputs can be established when there isn't a hidden layer for them to mutually tune and take as input/pass as input to other output nodes. Jan 13, 2020 at 4:22

You can do it with only one Neural Network. But your Neural Network should look like this:
Input Layer: 34 Nodes(one per your input column)
Output Layer: 8 Nodes(one per your output column)

You can add as many as and as big as hidden layers you want in the Neural Network. So Neural Network outputs 8 predicted values and each value will be a different regression of the inputs.

• which package of which language supports so?? Jun 14, 2018 at 12:27

I was wondering the same; here are my ideas:

I suppose that if the outputs share some hidden patterns, then training can benefit from simultaneously learning the regression for all the outputs.

It would be interesting to try an architecture where you build a neural network for each output, but all the neural networks share some layers (the first half layers for example). Then you could train each neural network at the same time: inside the learning loop, each neural network is trained one step (with one batch) sequentially.

This would be similar to knowledge transfer, but with the difference that in knowledge transfer each neural network is fully trained before reusing some part of it to train another neural network.