Regression to find optimal linear combination of ensemble of neural network weightings?

Question

I have N neural networks trained on different subsets of features of a dataset and with slightly different methods. My problem is a multiclass output, with the output layer comprising of the softmax output for each class.

The outputs for each input on every network look like:

array([ 0.05925509, 0.10252554, 0.72204741, 0.05106815, 0.00929179, 0.05581202])

With each element denoting the prediction 'pseudo-probability' of the network choosing that class

And I ofcourse have the actual labels of the form:

array([ 0, 0, 1, 0, 0, 0])

I want to use regression to find an optimal linear combination of these outputs for which to weight each network to in the ensemble model.

I want to find $\alpha_i$ where $Y = \sum{\alpha_i}X_i$ where $X_i$ is the prediction in the above format for the $ith$ network.

Would I need to use some sort of multi-dimensional regression here? Is it even valid to treat this as a classical regression problem since each 'feature' in my regression input is actually six dimensional?

Answer 1

You can use multivariate regression for this problem, without a problem.

But since you already use neural networks it is more natural to me to solve this problem using neural net with one neuron on the top of your outputs. $\alpha_i$ would be weights of this new neural net and you will not use activation function (i.e. use linear activation function).

Also, keep in mind that you will need $N$ different subset of features created from each new (unseen) example in order to perform prediction on unseen data.