2
$\begingroup$

I want to use an exponential decay model in python to relate the flow rate in a device to the mass left inside it, in particular $flow=a−b×e^{−c×mass}$ where a, b and c are the parameters of the model. Here are some of my measurements together with fitted curves.

enter image description here

enter image description here

For a lot of the experimental runs the data points at the start of the experiment would generally behave differently to the rest, which is considered to be due to how different materials begin to flow through the equipment. Sometimes there would be a streak of measurements that are systematically off the trend, other times there would be a megaphone shape with variability slowly decreasing to the level of the rest of the set. Very rarely there are some nonconforming measurements in the end of the experiment (low mass) that need to be trimmed off but usually they are fewer.

I want to truncate the data to a region on which the decay model will fit fairly consistently. Because I have a lot of experimental runs I'm essentially looking for some reasonable procedure to decide where is a good place to trim off the data from the right end and code this in python to automate data cleaning. I was wondering if someone could advise me on a method that I could use to chose where to trim the data.

One way that came to mind was to fit the model to trimmed data sets and compare the $r^2$ statistic for these models. To explore it visually I made heatmaps of the $r^2$ for different truncations to see if there is a pattern that could justify a trimming strategy. The green tickmarks on the right plots are showing every tenth data point (doesn't look great when there is a lot of data but still). For the cases I've looked at the recommendation of the plot seems to correspond to visual intuition.

enter image description here Peak at top left corner indicates no need to trim enter image description here

As this article suggest, we would expect the $r^2$ to drop as the range of the independent variable is decreased, but the improvement from trimming seems to dominate this effect. The $r^2$ statistic seems to peak at some truncation and then drop. Perhaps this indicates truncation based on $r^2$ is reasonable? Still it is my intuition that this may be a flawed approach as we base a model on the 'well-behaved' points only and thus we may end up with a model that is over-fitted and underestimate the range of masses for which we can make good predictions with the decay model.

Another idea that came to mind, particularly in the cases other than the megaphone shape, is to calculate cumulative mean and base a threshold on it, or perhaps perform clustering on the measurement pairs and truncate the smaller clusters.

$\endgroup$
1
  • $\begingroup$ Visually the data on the right-hand side of the scatterplots appears to approximate a straight line, except for the data to be truncated. My first idea was to linearly fit that data with OLS, and then either iteratively truncate to look for the point when the "straight line" R-squared stopped increasing, or else treat the to-be-truncated data more like a region with outliers to be removed. These methods would both have the effect of removing data points based on their fit characteristics, rather than on experimental characteristics as @Patrick notes in his answer. $\endgroup$ Oct 23, 2019 at 17:17

1 Answer 1

2
$\begingroup$

I believe this to be a bad idea: you are selecting your data based on the data fitting to your model and that can lead to overfitting and possible biases/inaccuracies in your inferences. Basically: how do you know a point is a measurement mistake vs something else just from the value of the point? The recommended practice would be to figure out some sort of indicator in your experiment that can tell you if a point is bad or to improve your experiment so that these bad values don't happen (I know this is easy for a stranger over the internet to say...). One approach would be to try and figure out the measurement uncertainties on each data point (it looks like you have it for some of them but not others) and then try a weighted fit. That would for example solve your problem in some of the cases. You could also experiment with some sort of weighting of the data based on the ease of measurements or something.

Ok with that pontification aside: There are two approaches I can think of:

1) Make it a piecewise model with a changepoint where you explicitly introduce a measurement error which is dependent on which side of the changepoint you are on. I would personally do this as a Bayesian model using Stan. Something like this: https://ecosang.github.io/blog/study/piecewise-regression-with-stan/ or https://janhove.github.io/analysis/2018/07/04/bayesian-breakpoint-model but instead of letting the parameters vary you let the measurement error vary. Specifically for the second link instead of having a slope_before and a slope_after you would have an error_before and error_after (currently that model just has one error parameter). You would also need to change the functional form of the model (right now it's fitting a piecewise linear regression). You can also change this to incorporate an offset instead of changing the measurement error (so an intercept_before and intercept_after). Heck you could let everything change with the changepoint. You get a lot of flexibility out of this approach.

2) You can fit a changepoint model (e.g., the bcp package in R) on the residuals of your fit and truncate the data after that changepoint and then refit the model. I don't like this approach as much because you're throwing the data out completely but it would solve your problem in an automated way.

$\endgroup$
2
  • $\begingroup$ Hi Patrick, thank you for the suggestion. I share your concerns about the over fitting but my intuition is that it would only cause trouble if I wanted to use my model on the truncated region as well. I am just interested in identifying one that conforms to a trend, and use the model for points in that region alone. I have some idea about what causes the cases where there's a streak of point deviating from the trend at the start, it's not about biased measurements but about the mechanics of the machine. I was thinking I may try to model this effect and your suggestions will be very helpful. $\endgroup$
    – st210
    Oct 23, 2019 at 14:34
  • $\begingroup$ In that case I would go with the changepoint analysis to reject data based on fitting the model once, finding the changepoint, and then refitting the model. Check out the bcp package in R or if you're using another language then I'm sure there are packages for that as well. See for example github.com/deepcharles/ruptures for python. $\endgroup$
    – Patrick
    Oct 23, 2019 at 14:57

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.