I'm trying to predict proportion data, and I've got a small dataset (~4000), so holding out a test and validation set isn't practical. However, bagging is practical because the cost of training isn't high.

I'm using Keras with a sigmoid (logistic) output. My model is effectively $$ y = g(f(x)) $$ where $g()$ is the sigmoid function, and $f(x)$ is a neural net. The top layer of the neural net, pre sigmoid, will be a real-valued linear combination of derived variables.

My question

In general: will I get better predictive performance forming estimates of $$ y^* = E_{bag}[g(f(x))] $$ or $$ y^* = g(E_{bag}[f(x))] $$

My instinct is that the latter would work better, but I'd be curious to know if this is (1) true, and (2) provably generally true.

NB: There is nothing neural-net specific about this question, but FWIW I am using neural nets because my input data is about as long as it is wide, and has a structure that is both nonlinear (elasticnet is out) and has useful, known structure (trees are out).

  • $\begingroup$ One is tempted to refer to Jensen's inequality, but note that $g$ is not convex so it does not apply here. $\endgroup$ – Sycorax Jun 15 '18 at 16:04
  • $\begingroup$ Indeed. And even if it was, it's not obvious how knowing which is bigger would tell you much about predictive performance. $\endgroup$ – generic_user Jun 15 '18 at 16:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.