Use of weights in non-linear least square fitting

Question

I would like to have your suggestions and help concerning my problem.

I have images generated on a position sensitive detector. The signal for each pixels corresponds to the amount of 'particles' interacting with the detector, so I would consider the signal distribution as a counting statistics.

I have to fit a profile of these images with a non-linear function and for this I am using a non-linear least square minimization (Levenberg-Marquardt algorithm) as implemented in scipy least_square.

So far, I have not used any weights for the input data (that corresponds to have a constant value of 1 for all points) and the fit is 'visually' looking very nice. On the other hand, the chi_square is enormously high and I think that this is due to the fact the I am not scaling the input data for their uncertainty.

My idea was to use the counting statistics to provide the uncertainty of each point. I was thinking to calculate the variance standard deviation of each pixel as the square root of its number of counts and then use it as weight for the fit. In this way, the pixels with the highest signal will have a higher variance standard deviation and a higher weight. On the contrary the 'almost' empty pixels will have a lower weight in the fit.

what do you think it is the most reasonable approach?

EDIT: after Sextus's answer

1. My fitting model function is not necessarily exact, meaning that it is the mathematical representation of the physical phenomenon occurring. Thus there is no reason why the calculated chi squared should follow the chi squared distribution.

2. So far, the obtained fits were reasonably well following the experimental distribution and the qualitative information obtained are well describing the phenomenon. But since I'm performing several thousands of fits, I was looking for a fit-goodness parameter to use to quickly identify failed and bad fits. After the remarks of @Sextus, I guess that the chi squared cannot be used for this purpose. Do you have any fit-goodness parameter to suggest under these circumstances?

3. For the moment, I'm not using any weights, that is equivalent to give to all points the same weight of 1. What will be the benefit of using weights? Will the uncertainty of the determined parameters get better? Will the fit-goodness parameter get better? For this purpose, can make sense to use the standard deviation of each pixel as weights? In this case, I would use the assumption that each pixel is independent and follows a Poissonian distribution with standard deviation = square root of the mean.

Thanks again for your help, toto

Answer 1

A nice reference relating to this question is

The conditions under which chi square measures the discrepancy between observation and hypothesis

Fisher RA 1924 Journal of Royal Statistical Society, 87, pp. 442-450

https://www.jstor.org/stable/2341149

In the second section "Reasons for abnormal distribution of $\chi^2$" Fisher gives three reasons

1. If the hypothesis tested is not in fact true

2. If the method of estimation employed is inconsistent

   E.g. if the model has a bias that remains for an infinite sample, like using binned values instead of continuous data values.

3. If the method of estimation employed is inefficient

   E.g. the terms for the chi squared $\frac{(\text{observed}-\text{expected})^2}{\text{expected}}$ may have $\text{expected}$ based on the estimates of a fitted model. When that model is not efficiënt the conputed chi squared will not approach a chi squared distribution.

In your situation you may have many ways that the first case applies.

• Your hypothesised model might be simply wrong. Possibly it can be close enough for practical purposes and you are happy with it. But a small discrepancy between the true model and your hypothetical model can make the chi squared value increase indefinitely (see also the discussions about testing for normality being often not needed).

• You may have miscalculated the variance. It is not entirely clear because you don't provide an exact calculation and method but something like the following doesn't seem right (it is the standard deviation is equal to the square root but not the variation)

  was thinking to calculate the variance of each pixel as the square root of its number of counts

  Also the error might not be just due to Poisson noise of counting, and it can also be that your count events are correlated ot that you have additional sources of error.

• If there are additional sources of error, then this error may also be correlated. Assuming that it is iid distributed error overestimates the degrees of freedom.

---

About your edit.

Do you have any fit-goodness parameter to suggest under these circumstances?

You can use the coefficient of determination which expresses the degree in which your model explains variations compared to a minimal model. In the case of non-normal distributed variables you can use a pseudo-R² measure.

For the moment, I'm not using any weights, that is equivalent to give to all points the same weight of 1.

It is unclear what sort of regression you are doing. The exact method is not given in the question. If you are not yet using a generalized linear model, a method which includes weights for the different data points, then you could start with that.