I have N neural networks trained on different subsets of features of a dataset and with slightly different methods. My problem is a multiclass output, with the output layer comprising of the softmax output for each class.

The outputs for each input on every network look like:

array([ 0.05925509, 0.10252554, 0.72204741, 0.05106815, 0.00929179, 0.05581202])

With each element denoting the prediction 'pseudo-probability' of the network choosing that class

And I ofcourse have the actual labels of the form:

array([ 0, 0, 1, 0, 0, 0])

I want to use regression to find an optimal linear combination of these outputs for which to weight each network to in the ensemble model.

I want to find $\alpha_i$ where $Y = \sum{\alpha_i}X_i$ where $X_i$ is the prediction in the above format for the $ith$ network.

Would I need to use some sort of multi-dimensional regression here? Is it even valid to treat this as a classical regression problem since each 'feature' in my regression input is actually six dimensional?


1 Answer 1


You can use multivariate regression for this problem, without a problem.

But since you already use neural networks it is more natural to me to solve this problem using neural net with one neuron on the top of your outputs. $\alpha_i$ would be weights of this new neural net and you will not use activation function (i.e. use linear activation function).

Also, keep in mind that you will need $N$ different subset of features created from each new (unseen) example in order to perform prediction on unseen data.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.