Neural network for multiple output regression

Question

I have a dataset containing 34 input columns and 8 output columns. One way to solve the problem is to take the 34 inputs and build individual regression model for each output column. I am wondering if this problem can be solved using just one model particularly using Neural Network.

I have used a multilayer perceptron, but that needs multiple models just like linear regression. Can sequence to sequence¹ learning be a viable option? I tried using TensorFlow it does not appears to be able handle float values.

Any suggestion to tackle this problem by using just one unified model specially using neural network will be appreciated.

1. Ilya Sutskever, Oriol Vinyals, & Quoc V. Le (2014). Sequence to Sequence Learning with Neural Networks. Advances in Neural Information Processing Systems, 27. (pdf)

Answer 1

At first I thought generic_user's comment was a show-stopper, but I just realized it isn't:

If I train d different networks on d different outputs, then each one will be fit to that dimension with no regard for the others.

But if I train one network with d outputs and use all outputs for backpropagation, then each weight in every layer in the network will be adjusted so that all d outputs are more accurate.

That is: Each network parameter will be adjusted by a sum of gradients (how each output varies with a "wiggle" in this parameter) such that adjusting it in the chosen up or down direction causes more accurate output overall--even if adjusting the weight that way causes some dimensions of the output to be less accurate.

So, yes, the thing that ultimately distinguishes each output is just a linear equation encoded in the last layer, but in training one multi-output network every layer will become better at presenting that last layer with something that allows it to do its job better. And so it follows that the relationships between outputs will be accounted for by this architecture.

You might be able to do better than a fully-connected net by making your architecture reflect any known relationships in the output, just as deep networks do better than shallow ones by exploiting "compositional" relationships between inputs.

Answer 2

A neural net with multiple outcomes takes the form $$ \mathbf{Y} = \mathbf{\gamma} + \mathbf{V}_1\Gamma_1 + \epsilon\\ \mathbf{V}_1 = a\left(\gamma_2 +\mathbf{V}_2\Gamma_2\right)\\ \mathbf{V}_2 = a\left(\gamma_3 +\mathbf{V}_3\Gamma_3\right)\\ \vdots \\ \mathbf{V}_{L-1} = a\left(\gamma_L+ \mathbf{X}\Gamma_L\right)\\ $$ If your outcome has the dimension $N\times 8$, then $[\gamma_1, \Gamma_1]$ will have the dimension $(p_{V1}+1) \times 8$.

Which is to say that you'd be assuming that each outcome shares ALL of the parameters in the hidden layers, and only has different parameters for taking the uppermost derived variable and relating it to the outcome.

Is this a realistic assumption for your context?

Answer 3

You can do it with only one Neural Network. But your Neural Network should look like this:
Input Layer: 34 Nodes(one per your input column)
Output Layer: 8 Nodes(one per your output column)

You can add as many as and as big as hidden layers you want in the Neural Network. So Neural Network outputs 8 predicted values and each value will be a different regression of the inputs.

Answer 4

I was wondering the same; here are my ideas:

I suppose that if the outputs share some hidden patterns, then training can benefit from simultaneously learning the regression for all the outputs.

It would be interesting to try an architecture where you build a neural network for each output, but all the neural networks share some layers (the first half layers for example). Then you could train each neural network at the same time: inside the learning loop, each neural network is trained one step (with one batch) sequentially.

This would be similar to knowledge transfer, but with the difference that in knowledge transfer each neural network is fully trained before reusing some part of it to train another neural network.

I bet someone has thought about this before, but I have no reference to it.