I want to use an exponential decay model in python to relate the flow rate in a device to the mass left inside it, in particular $flow=a−b×e^{−c×mass}$ where a, b and c are the parameters of the model. Here are some of my measurements together with fitted curves.
For a lot of the experimental runs the data points at the start of the experiment would generally behave differently to the rest, which is considered to be due to how different materials begin to flow through the equipment. Sometimes there would be a streak of measurements that are systematically off the trend, other times there would be a megaphone shape with variability slowly decreasing to the level of the rest of the set. Very rarely there are some nonconforming measurements in the end of the experiment (low mass) that need to be trimmed off but usually they are fewer.
I want to truncate the data to a region on which the decay model will fit fairly consistently. Because I have a lot of experimental runs I'm essentially looking for some reasonable procedure to decide where is a good place to trim off the data from the right end and code this in python to automate data cleaning. I was wondering if someone could advise me on a method that I could use to chose where to trim the data.
One way that came to mind was to fit the model to trimmed data sets and compare the $r^2$ statistic for these models. To explore it visually I made heatmaps of the $r^2$ for different truncations to see if there is a pattern that could justify a trimming strategy. The green tickmarks on the right plots are showing every tenth data point (doesn't look great when there is a lot of data but still). For the cases I've looked at the recommendation of the plot seems to correspond to visual intuition.
As this article suggest, we would expect the $r^2$ to drop as the range of the independent variable is decreased, but the improvement from trimming seems to dominate this effect. The $r^2$ statistic seems to peak at some truncation and then drop. Perhaps this indicates truncation based on $r^2$ is reasonable? Still it is my intuition that this may be a flawed approach as we base a model on the 'well-behaved' points only and thus we may end up with a model that is over-fitted and underestimate the range of masses for which we can make good predictions with the decay model.
Another idea that came to mind, particularly in the cases other than the megaphone shape, is to calculate cumulative mean and base a threshold on it, or perhaps perform clustering on the measurement pairs and truncate the smaller clusters.