Context and goals:

Consider $x$ a scalar (deterministic) independent variable and $y = f(x;\beta)+\eta$ be a dependent random variable obtained trough $f$ with some parameters $\beta$ and $\eta$ is some noise modeled as a Gaussian with zero mean and unknown variance. Concretely, chose $$ f(x;\beta) = \beta_1 + \frac{\beta_2}{1+e^{\beta_3x}} $$ Now, assume that we made $M$ experiments (with $M$ presumably low) and obtained $M$ pairs of measurements $(x_1,y_1),\dots,(x_M,y_M)$.

  • Question: Given the data, How to obtain a way to estimate of $y$ for any $x$ as well as its "confidence"?

Let me explain my attempt to solve this problem:

My idea was to fit the parameters to the data using nonlinear regression by means of optimization (gradient descent) for example. Thus obtain an approximate model $f(\bullet;\hat{\beta})$. Then, for certain value of $x$ we can model that its corresponding $y$ is a Gaussian centered at $f(\bullet;\hat{\beta})$ (right?). However, since I don't know the variance of the noise $\eta$, the most we can do is to compute the sample variance of $y_1,\dots,y_M$ and use a student's-t distribution instead.

The problem is that, I'm not totally convinced that $f(\bullet;\hat{\beta})$ can be used as the mean of $y$. I guess one has to check if $f(\bullet;\hat{\beta})$ is an unbiased estimate of $y$. However, I don't know how to check that. Moreover, by using the sample variance I only get a number which represents the "confidence" for all $y$ which seems unsatisfactory to me since I would expect to be more confident of values of $y$ for $x$ near the samples $x_1,\dots,x_M$. But maybe my intuition is not correct.


Given the nonlinear regression model $$ Y_{i} = f( x_{i}, \underline{\beta} ) + \eta_{i} \quad \textrm{where} \quad \eta_i \overset{\text{iid}}{\sim} \mathcal{N}(0, \sigma^{2}) $$

and given the data $$ (x_{i}, y_{i}), i = 1, ..., M $$ a standard approach to estimate the unknown parameters $\underline{\beta} = (\beta_{1}, ..., \beta_{p})$, is to apply the method of least squares by minimizing the sum of squared residuals $$ \sum_{i=1}^{M} \left( y_{i} - f(x_{i}, \underline{\beta}) \right)^{2} $$ Since this problem is not linear in the unknowns $\beta_{1}, ..., \beta_{p}$, the estimation can not be derived explicitly as in linear regression, but we can solve it with the help of an iterative numeric procedure such as Gauss-Newton method, see: http://en.wikipedia.org/wiki/Gauss-Newton_method.

The least squares estimator $\hat{\beta}$ is asymptotically multivariate normally distributed with mean $\beta$ and therefore unbiased.

For a given $x_{0}$, the function $f$ can be approximated with: $$ f(x_{0}, \underline{\hat{\beta}}) \approx f(x_{0}, \underline{\beta}) + \frac{\partial f(x_0, \underline{\beta}) }{ \partial \underline{\beta}} (\underline{\hat{\beta}} - \underline{\beta}) $$ and the approximate $(1 - \alpha)$ confidence interval for the function value $f(x_{0}, \underline{\beta})$ is then $$ f(x_{0}, \underline{\hat{\beta}}) \pm q_{1-\alpha/2}^{t_{m - p}} \cdot \textrm{se}( f(x_{0}, \underline{\hat{\beta}}) ) $$

We can estimate the expected value $E(Y_{0}) = f(x_{0}, \underline{\beta})$ with $f(x_{0}, \underline{\hat{\beta}})$. And to get the interval where the observations will lie with high probability we also have to take into account the variance of the error $\eta_{0}$ $$ f(x_{0}, \underline{\hat{\beta}}) \pm q_{1-\alpha/2}^{t_{m - p}} \cdot \sqrt{ \hat{\sigma}^{2} + \textrm{se}( f(x_{0}, \underline{\hat{\beta}}) )^{2} } $$ This is called prediction interval or forecast interval.

  • $\begingroup$ This is potentially what I was looking for. Just wanted to know if you could provide a reference were I can look at this in more detail. Thanks sir $\endgroup$ Nov 16 '20 at 17:19
  • $\begingroup$ Sure, great introductory notes are available here: stat.ethz.ch/~stahel/courses/cheming/nlreg10E.pdf They are based mostly on the book: Nonlinear Regression Analysis & Its Applications by Bates et. al. $\endgroup$ Nov 16 '20 at 17:46
  • $\begingroup$ awesome reference, this is precisely what I was looking for. Thanks! $\endgroup$ Nov 16 '20 at 17:51

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.