Learning to predict maximum of parameterized function class I am interested in a multi-task regression problem: I have a parametrized function $f_x : \mathcal{R}^n -> \mathcal{R}$ where $x \in \mathcal{R}$ is a real-valued parameter. For some values of $x$, I have a large set of (noise-less) training examples. I would like to learn to predict for some new $x'$ where the maximum of $f_{x'}$ lies. $f_x$ and $f_{x'}$ should be similar if $x'$ is close to $x$. The $f_x$ are not restricted to any parametric function class but are upper-bounded by $0$. 
As an add-on, it would also be very nice if not only the estimate of the maximum but also some uncertainty-measure would be available. Think of the estimated maximum of $f_{x'}$ as a starting point for a black-box optimization of $f_{x'}$.
My first idea was to treat the entire problem as a single regression problem $f: \mathcal{R} \times \mathcal{R}^n -> \mathcal{R}$. However, this is not very amenable for estimating the maximum of $f_{x'}$.
An other option would be to learn a mapping $g : \mathcal{R} -> \mathcal{R}^n$, which maps task parameter to the estimated maximum of $f_{x'}$, i.e., $g(x') = \arg\max\limits_{y \in \mathcal{R}^n} f_{x'}(y)$. This could be trained based on the pairs $(x, \arg\max\limits_{y \in \mathcal{R}^n} f_{x}(y))$ for the $x$ for which training data exists. However, it would loose all information contained in the non-maximal datapoints.
Any suggestions?
 A: If you believe that the functions vary "smoothly" according to the parameters, I think your idea of treating it as a single regression problem should work. You can fit a function $f: \mathcal{R} \times \mathcal{R}^n \rightarrow \mathcal{R}$ using say a Gaussian Process Regression (GPR aka Kriging), and then maximize the $n$-argument function obtained by fixing the parameter to $x'$ (the value that you are interested in).
The GPR model will give you uncertainties as a function of the $n+1$ parameters of function $f$. Essentially you'll have a (Gaussian) error band around the value of $f$ at any point in the $n+1$ dimensional space. So, in principle, you can draw several samples of $f$ at the fixed parameter value $x'$ and maximize each one separately. (Each sample will be a function of $n$ variables.) The variability in the different maxima will give you an estimate of the uncertainty.
The sampling of a function would be hard to do in general, but you only need to sample it at the points where the blackbox optimizer will evaluate it. So you can generate samples of the function values according the posterior distribution given by the GPR in a sequential manner, when the optimizer demands a function value.
