Is chaining neural networks in this way a good way to estimate a prediction interval?

Question

Suppose you want to predict the outcome of some real valued function $f$. The details of the function are unknowable and it also has a stochastic component. You identify some variables $\theta$ which you think could help to predict $f$.

You suspect that $f$ is non linear in $\theta$ so you decide to implement a neural network $\gamma_A$. So your input layer is $\theta_1; \theta_2;...;\theta_j$, you have a single output $\hat{f}$, you have $n$ observations in your training set and your loss function is $\sum_{i=0}^n(f_i-\hat{f_i})^2$.

You notice that your neural network performs quite well for certain ranges of $\theta$ but not so well for other ranges.

You decide that you need a prediction interval for $\hat{f}$ given $\theta$ so you decide to implement another neural network $\gamma_B$ to estimate this prediction interval. The input layer for $\gamma_B$ is $\hat{f};\theta_1; \theta_2;...;\theta_j$, you have a single output $\sigma_\hat{f}$ and your loss function is $(\frac{\sum_{i=1}^n1_{(\hat{f}-\sigma_\hat{f},\hat{f}+\sigma_\hat{f})}}{n}-\lambda)^2$, where $\lambda$ is a parameter you control. For example, for $\lambda = 0.5$, the loss function will be minimised if 50% of observations fall with the range $(\hat{f}-\sigma_\hat{f},\hat{f}+\sigma_\hat{f})$.

My questions are

1. Is chaining the neural networks in this way a good idea? Am I dramatically increasing the risk of over-fitting if I do this?
2. Assuming the answer to 1 is "It's OK to do this", is there a better loss function for $\gamma_B$?
3. Is there some other method of estimating a prediction interval, conditional on $\hat{f}$ and $\theta$?

Answer 1

A more obvious thing that is done frequently (e.g. in Kaggle competitions that have featured prediction interval coverage in their scoring metric) is to

1. have more than one layer with non-linear activation functions (e.g. ReLU), which leads to richer representations/more complex functional space (e.g. the proposal above cannot capture interactions between inputs),
2. output both the point estimate and the prediction interval limits from the same neural network, you can achieve that by firstly outputting the estimate as usual and two quantities (or one, if you want to impose equal-length of the intervals in both directions) that enter an activation that goes from $(-\infty, \infty)$ to $(0, \infty)$ (e.g. exponential) and give how far the interval limits are below and above the estimate (quantile regression techniques, e.g. pinball loss are options for the loss function here).

You just need to balance the trade-off between interval coverage and estimation loss. Doing this through a single network has the potential to gain on each task through multi-task learning (which may lead to better internal representations of the data) and seems to usually do better than a multi-step process.