Why does a Least squares distribution look like a parabola?

Question

For regression analysis, one often uses the least squares method to minimize the quadratic differences between data and a model function f as follows:

$$\chi^2=\sum_i \frac{(\text{data}\, _i-f_i(p))^2}{\sigma_i^2}$$

with the parameter to be fitted $p$ (see question for a plot and discussion).

Why (and under which assumptions) does $\chi^2(p)$ look like a parabola?

In my work, due to computational intensity (of $f$), I can't use common tools to find the minimum and have a small number of data points, which is why I then fit a parabola to the $\chi^2(p)$ distribution to find the estimated minimum that way.

Edit1: In my case, $\text{data}\, _i$ and $f_i$ are produced by the same numerical procedure (I do sensitivity tests of parameters), which is why I can assume that the two distributions look very similar for small variations of $p$.

Answer 1

If $f(p) $ is a linear function of the parameters you'll get a parabola, since we are squaring it.

As for the why, we are looking for something whose solution is easy to find and possibly can be found explicitly and assigning positive weight to any difference between data and model. We could use the absolute value, but it then becomes annoying to minimize.