# Optimization landscape and relationship to parameter uncertainties

In linear/nonlinear regression we try to find the minimum for the sums of squares as a function of the model parameters. For a one dimensional system this is just a simple 2D curve. Parameter on the x axis, sums of squares on y axis.

I was under the impression that estimates for uncertainties in the parameter estimates were related to the shape of the landscape formed from the sum of squares as a function of the parameters. Eg I thought that as a data set became more noisy the shape near the optimum would become flatter indicating less certainty in the parameter estimates. I tried a simple one parameter linear model, y = p x, plotting the sums of squares as a function of the parameter using synthetic data. What I got was a nice looking parabola as expected. With no noise in the data the parabola minimum is exactly at the correct parameter value. I thought that as I added noise to the data the parabola would start to widen out (and also move upwards which it did) indicating less certainty. My thinking was that the parameter uncertainties are related to the curvature (hessian) hence I would see that reflected in the shape of the sums of squares landscape. But I’m not seeing any significant change in the overall shape of the curve. Obviously I’m misunderstanding this. My question is what’s the intuition with respect to parameter uncertainties and errors in the data with respect to the sums of squares landscape?

• the hessian is just the covariance matrix of the inputs.the parameter uncertainty is multiplied by the noise variance. max.pm/posts/hessian_ls Commented Oct 13, 2023 at 0:12
• I know the formula, I don’t understand how it works, ie the intuition. My main question is what’s the relationship between the parameter uncertainties and the landscape formed by the optimization algorithm. I thought there was one but I’m not sure now.
– user36563
Commented Oct 13, 2023 at 0:35

parameter uncertainty (variance) = noise variance x curvature of error surface. hessian of LS

The curvature of the error surface is given by the covariance matrix of the inputs.

so

1. uncertainty is proportional to noise variance.

2. furthermore it is inversely proportional to the input variance ( if I double an input, the coefficient will halve and similary the coefficient uncertainty).

3. Parameter uncertainty will also be inverserly affected by the absolute correlation (if 2 inputs are perfectly correlated then the uncertainty is 'infinite', since I can set coefficient $$\beta_1$$ to any value by appropriately compensating with $$\beta_2$$

• How are the uncertainties related to the shape of the LS surface? Does noise make the bottom on the surface flatter? My experiments with a single parameter fit seems to suggest noise make no difference to the shape of the LS surface.
– user36563
Commented Oct 16, 2023 at 17:49
• "the curvature of the error surface is given by the covariance matrix of the inputs." this means that the curvature of the error surface doesnt depend on the noise Commented Oct 16, 2023 at 18:35
• Would it be fair to say that the only thing that contributes to the shape of the LS landscape are things like missing data, non-identifiability and correlations between parameters inthe model?
– user36563
Commented Oct 17, 2023 at 19:05
• After reading more I am still a bit uncertain. A one standard deviation change in a given parameter is one chi-square above the LS minimum. Doesn't this suggest that the shape of the ellipse contours around the LS minimum says something about the uncertainty? If there is noise in the data I assume there will be more uncertainty in the estimated parameter values. This suggests the ellipse will have wider contours in order to reflect the fact that the standard deviation will be larger in order to achieve a one chi-square change? If you think this should be posted as a new question I can do that.
– user36563
Commented Oct 17, 2023 at 19:28
• distance travelled = (average) speed x time. speed doesn't depend on time travelled yet your distance travelled will depend on time. parameter uncertainty = noise variance x curvature of error surface. so the noisier your dependent variable, the greater the parameter uncertainty even though the curvature of the error surface doesn't change. Commented Oct 18, 2023 at 6:45