I'm trying to fit data to a line $y=ax+b$ (you can download it in this gist). You can see a fit in the next image:

Fit of the data

To do so, I'm using analytical methods of minimization of the cost function, defined as the sum of squared residuals (like this one) instead of numerical ones. The coefficients I obtain agree well with numerical software, but the standard deviations computed for them are a bit strange when I check the cost function. If I understand the theory well, the cost function should follow a $\chi^2$ distribution of mean $N$ (in this case $N=100$, the number of datapoints) and variance $2N$.

The cost function in the obtained parameters should be $\sim 100$, and at one standard deviation of the cost function (at $\sqrt{200}$ from the minimum value) should lie the parameters plus one standard deviation.

In the next image you can see a contour plot of the cost function, where I have marked some lines corresponding to $c_{\text{min}} + 0.5\sqrt{2c_{\text{min}}}/2$ , $c_{\text{min}} + \sqrt{2c_{\text{min}}}/2$, etc. where $c_{\text{min}}$ is the minimum of the cost function ($\sim 115$), which should give me the deviations of the parameters at $0.5\sigma$, $1\sigma$, etc. :

Cost function

In yellow, I have marked my obtained parameters, with their mean and their standard deviations, which as you can see are way smaller. Is this expected? Do you see something that could be wrong?

  • $\begingroup$ Now that that is cleared up, could you explain the connection you are expecting between the "cost function" (which I presume is the sum of squared residuals) and the standard errors of the coefficient estimates? $\endgroup$
    – whuber
    Commented Dec 1, 2016 at 21:30
  • $\begingroup$ I expect that the cost function at one standard deviation gives me the same estimate of the standard deviation of the parameters as the analytical method. In graphical terms, I expect that the orange lines just touch the $1\sigma$ contour of the cost function. $\endgroup$ Commented Dec 1, 2016 at 21:52


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.