A book I have on regression analysis describes a technique for determining confidence contours of the parameters of a linear model

$$ Y^{\textrm{model}} = f(\boldsymbol{x}, \boldsymbol{\theta}) = \boldsymbol{\theta}^T\boldsymbol{\phi} $$

where Y (scalar) is the dependent variable, $\boldsymbol{\theta}$ is the vector of $p$ parameters, and $\boldsymbol{\phi}$ is a vector where every element is some function of the independent variable vector $\boldsymbol{x}$.

For a set of $n$ observations, the error sum of squares is defined as

$$ S(\boldsymbol{\theta}) = \sum_{i=1}^n \left ( Y_i^{\textrm{true}} - Y_i^{\textrm{model} } \right )^2 = \left ( Y_i^{\textrm{true}} - f(\boldsymbol{x}_i, \boldsymbol{\theta}) \right )^2 $$

The least squares estimate for the parameters is $\hat{\boldsymbol{\theta}}$, which minimizes the error sum of squares $S$.

In parameter space (the $p$-dimensional space where every axis corresponds to one of the parameters) contour surfaces can be defined for specific values of the error sum of squares $S$. For linear models like $f$ these contour surfaces will be (hyper)ellipsoids centered around the least squares estimate $\hat{\boldsymbol{\theta}}$.

If error $\epsilon$ around model predictions is normally distributed with a mean of zero

$$ \epsilon \sim N(0,\sigma^2) $$

then a value for $S$ can be found that corresponds to a $1-\alpha$ "confidence contour". Similar to a confidence interval, it can be stated with $1-\alpha$ confidence that the point corresponding to the true values of the model parameters $\boldsymbol{\theta}^{\textrm{true}}$ is enclosed by this confidence contour.

The equation defining the confidence contour is

$$ S(\boldsymbol{\theta}) = S(\hat{\boldsymbol{\theta}}) \left [ 1 + \frac{p}{n-p} F(p,n-p,1-\alpha) \right ] $$

where $F$ is the cumulative distribution function for the F-distribution. A confidence contour is depicted below for the example case when the linear model has $p=2$ parameters.

How can this technique be generalized for situations where there are multiple dependent variables of widely varying magnitudes?

  • $\begingroup$ After some further reading, it appears this question may be a replicate of 303204. $\endgroup$
    – Jack Elsey
    Commented Sep 6, 2019 at 16:27
  • $\begingroup$ My hunch is that the error sum of squares can also be summed across the number of dependent variables if the dependent variables are scaled so that all residuals have the same variance $\sigma^2$. $\endgroup$
    – Jack Elsey
    Commented Sep 7, 2019 at 0:09
  • $\begingroup$ But what if there are a lot fewer measurements for one of the dependent variables? $\endgroup$
    – Jack Elsey
    Commented Sep 7, 2019 at 2:50


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.