What is a sensible way to truncate data to a region that fits a model?

Question

I want to use an exponential decay model in python to relate the flow rate in a device to the mass left inside it, in particular $flow=a−b×e^{−c×mass}$ where a, b and c are the parameters of the model. Here are some of my measurements together with fitted curves.

For a lot of the experimental runs the data points at the start of the experiment would generally behave differently to the rest, which is considered to be due to how different materials begin to flow through the equipment. Sometimes there would be a streak of measurements that are systematically off the trend, other times there would be a megaphone shape with variability slowly decreasing to the level of the rest of the set. Very rarely there are some nonconforming measurements in the end of the experiment (low mass) that need to be trimmed off but usually they are fewer.

I want to truncate the data to a region on which the decay model will fit fairly consistently. Because I have a lot of experimental runs I'm essentially looking for some reasonable procedure to decide where is a good place to trim off the data from the right end and code this in python to automate data cleaning. I was wondering if someone could advise me on a method that I could use to chose where to trim the data.

One way that came to mind was to fit the model to trimmed data sets and compare the $r^2$ statistic for these models. To explore it visually I made heatmaps of the $r^2$ for different truncations to see if there is a pattern that could justify a trimming strategy. The green tickmarks on the right plots are showing every tenth data point (doesn't look great when there is a lot of data but still). For the cases I've looked at the recommendation of the plot seems to correspond to visual intuition.

As this article suggest, we would expect the $r^2$ to drop as the range of the independent variable is decreased, but the improvement from trimming seems to dominate this effect. The $r^2$ statistic seems to peak at some truncation and then drop. Perhaps this indicates truncation based on $r^2$ is reasonable? Still it is my intuition that this may be a flawed approach as we base a model on the 'well-behaved' points only and thus we may end up with a model that is over-fitted and underestimate the range of masses for which we can make good predictions with the decay model.

Another idea that came to mind, particularly in the cases other than the megaphone shape, is to calculate cumulative mean and base a threshold on it, or perhaps perform clustering on the measurement pairs and truncate the smaller clusters.

Answer 1

I believe this to be a bad idea: you are selecting your data based on the data fitting to your model and that can lead to overfitting and possible biases/inaccuracies in your inferences. Basically: how do you know a point is a measurement mistake vs something else just from the value of the point? The recommended practice would be to figure out some sort of indicator in your experiment that can tell you if a point is bad or to improve your experiment so that these bad values don't happen (I know this is easy for a stranger over the internet to say...). One approach would be to try and figure out the measurement uncertainties on each data point (it looks like you have it for some of them but not others) and then try a weighted fit. That would for example solve your problem in some of the cases. You could also experiment with some sort of weighting of the data based on the ease of measurements or something.

Ok with that pontification aside: There are two approaches I can think of:

1) Make it a piecewise model with a changepoint where you explicitly introduce a measurement error which is dependent on which side of the changepoint you are on. I would personally do this as a Bayesian model using Stan. Something like this: https://ecosang.github.io/blog/study/piecewise-regression-with-stan/ or https://janhove.github.io/analysis/2018/07/04/bayesian-breakpoint-model but instead of letting the parameters vary you let the measurement error vary. Specifically for the second link instead of having a slope_before and a slope_after you would have an error_before and error_after (currently that model just has one error parameter). You would also need to change the functional form of the model (right now it's fitting a piecewise linear regression). You can also change this to incorporate an offset instead of changing the measurement error (so an intercept_before and intercept_after). Heck you could let everything change with the changepoint. You get a lot of flexibility out of this approach.

2) You can fit a changepoint model (e.g., the bcp package in R) on the residuals of your fit and truncate the data after that changepoint and then refit the model. I don't like this approach as much because you're throwing the data out completely but it would solve your problem in an automated way.