# Bagging of models with link functions

I'm trying to predict proportion data, and I've got a small dataset (~4000), so holding out a test and validation set isn't practical. However, bagging is practical because the cost of training isn't high.

I'm using Keras with a sigmoid (logistic) output. My model is effectively $$y = g(f(x))$$ where $g()$ is the sigmoid function, and $f(x)$ is a neural net. The top layer of the neural net, pre sigmoid, will be a real-valued linear combination of derived variables.

My question

In general: will I get better predictive performance forming estimates of $$y^* = E_{bag}[g(f(x))]$$ or $$y^* = g(E_{bag}[f(x))]$$

My instinct is that the latter would work better, but I'd be curious to know if this is (1) true, and (2) provably generally true.

NB: There is nothing neural-net specific about this question, but FWIW I am using neural nets because my input data is about as long as it is wide, and has a structure that is both nonlinear (elasticnet is out) and has useful, known structure (trees are out).

• One is tempted to refer to Jensen's inequality, but note that $g$ is not convex so it does not apply here. – Sycorax Jun 15 '18 at 16:04
• Indeed. And even if it was, it's not obvious how knowing which is bigger would tell you much about predictive performance. – generic_user Jun 15 '18 at 16:06