# Correct error estimation for linear fit

This may be a simple problem, but I want to be thorough in setting up my problem as I'd like to know why I should proceed in one of two ways (or another if someone thinks it is suitable), so please bare with me.

I'm trying to calibrate the position of an array of 18 avalanche photodiode (APD) detectors. In the experiment, the signal is approximated to come along a single x-axis. To calibrate this, I have an LED that I can move along the axis into my experiment and I can see the intensity increase in a single APD channel as it goes past.

I can only get my LED to go past channel 3-12 of my array, but based on previous experiments, where the experimental setup isn't expected to have changed too much, I expect the signal each APD detects to be evenly spaced.

I have a set of x coordinates that I was able to measure, each with an associated error with position measurements when setting up the calibration, as well as a finite width of the signal detected in a single APD channel.

I use these errors to fit a weighted linear function to the data using scipy.optimize.curve_fit. So as a function of channel number, $$n$$, I can calculate the position, $$x$$, from the simple formula

$$x = a \times n + b$$

where $$a$$ and $$b$$ are the coefficients given out from the fitting function. I can also get out the co-variance matrix, and I can also calculate the mean square error (MSE) of the fit to my data points.

Question: how should I estimate the error of the position of the channels?

1) Combine the error of the two fit parameters $$a$$ and $$b$$ in quadrature, given as the diagonals of the co-variance matrix.

2) Alternatively, quoting the MSE as the error on any reading.

My error will be larger through method 1, and it seems more robust, especially if I want to use the fit to extrapolate the channels I wasn't able to reach with the LED. My instinct is to add the error of the coefficients in quadrature, but if that's so, what is the MSE actually good for? Or is it just a non-normalised number that gives me an indication of the goodness of the fit to the range I have?

• If any portion of the target audience is not trained in statistics, you should additionally include more direct metrics such as both the "average size of position error" and the "maximum size of position error". These are easier for people to understand as these can be directly visualized, they have units of length, and they are easier to discuss. – James Phillips Aug 15 '19 at 19:45
• So this is just calibration for the position of the APD channels, so ideally I only want to include errorbars in the x position, which will be the x axis in future plots I make, when plotting density or temperature etc. against position. So my question still stands, is the MSE and acceptable error, or should I do the full analysis? What is the MSE actually telling me? – Steven Thomas Aug 15 '19 at 21:17
• The Root Mean Squared Error (RMSE) acts like an average magnitude of error. Because "absolute value" has no continuous derivative, averaging the absolute value of the errors was intellectual horror in the early days - however you could effectively do the same thing by squaring the errors to make them all positive, finding the mean (average) value of those squares, and then take the square root of that mean value. That is where RMSE comes from. By itself, Mean Squared Error (MSE) does not inform directly. – James Phillips Aug 15 '19 at 22:51
• Thanks for your reply James. So if I understand you correctly, you're saying I shouldn't use the MSE to estimate an error for my position I'm reading off the graph, correct? – Steven Thomas Aug 16 '19 at 7:26
• I know what RMSE would tell you. What would MSE tell you? – James Phillips Aug 16 '19 at 11:53