5
$\begingroup$

I'm having a dataset where I measured the growth of a given molecule over 7 timepoints. The growth can be described by the formula:

$(P_{0} - plateau) * e^{-k t} + plateau$ , where k is the growth constant and $t$ the time. I want to estimate $P_{0}$, $plateau$ and $k$

All values are normalized to be between 0 and 1.

Due to the cost of the experiment, I only have one (depending on the perspective maybe 2) measurements per timepoint, however I have measurements for several thousands of molecules.

From what I have found out, the $R^{2}$ is not a good measure of fit in this case, despite it being used in many other studies I found. I also considered using the chi square or chi square reduced value, which I obtain when I fit my model using the python lmfit library with Levenberg-Marquardt or Least square minimization (https://lmfit.github.io/lmfit-py/fitting.html). However, I'm not sure how to interprete this value in terms what would be a good fit and what not. Also, according to wikipedia, I need to scale my chi square value, by a variance, which is not done in lmfit as far as I understand:

$\chi^{2}=\sum_{i}{\frac{(O_{i} - C_{i})^{2}}{\sigma^{2}_{i}}}$

I assume in my case, O would be the values I measured, and C the values from my fit. However I'm unsure what my variance would be in this case?

Here is an example plot of the data with a fit using the model and method described above:

Example fit with the model mentioned above

Can somebody tell me if I'm on the right path and or if there are any alternatives for estimating the quality of my fits relative to the other fits using the same model (maybe BIC or AIC)? Also, if there is any information missing I'm happy to add these. I'm happy about any hints which would put me in the right direction!

Improve this question
$\endgroup$
5
  • $\begingroup$ Your formula and plot looks more like shrinking than growth. $\endgroup$
    – Sextus Empiricus
    Sep 9, 2022 at 12:45
  • $\begingroup$ So to make sure I understand: you can produce many more plots like the one you've shown us, which each correspond to a given molecule, right? Do you always measure at the same timepoints, or does the place and number of measurements vary? $\endgroup$
    – John Madden
    Sep 9, 2022 at 13:07
  • $\begingroup$ @JohnMadden Yes, I can produce a couple thousand of theses plots for different molecules, the timepoints never change $\endgroup$
    – Manuel
    Sep 13, 2022 at 11:12
  • $\begingroup$ @SextusEmpiricus And yes, in this case its shrinking, there will also be growing curves. This depends on P0 and plateau, which are swapped for shrinking and growing curves $\endgroup$
    – Manuel
    Sep 13, 2022 at 11:14
  • $\begingroup$ In that case, @Manuel, I would evaluate fit visually, by plotting the error and bias of different models at those timepoints, averaged over the different molecules. This will show you what part is fit best/worst by individual models, giving you much more information than a simple "goodness of fit" statistic (these things are rarely used by today's statistician). $\endgroup$
    – John Madden
    Sep 13, 2022 at 12:25

2 Answers 2

Reset to default
2
+100
$\begingroup$

I'd leave $R^2$ out since your nonlinear models under comparison may have a different number of parameters. I would, instead, rely on one of these three approaches:

(a) use of information criteria such as BIC or AIC to select the best model or

(b) goodness of fit (GOF) test

(c) paired two-sample test.

In (a) you compute the AIC (or BIC) for each model and, then you choose as the 'best model' that with the lowest AIC (or BIC).

In (b) essentially you have to choose a statistic that measures either the fit of the model or the difference in fit between two models and compare it's observed value against a distribution. There is huge literature here, see Goodnessof-Fit Techniques, edited by Ralph B. D'Agostmo and Michael A, Stephens.

(c) is more heuristic, still kind of GOF, and goes as this. Suppose you want to compare two models $M_1, M_2$. Use your preferred estimation method and get the fitted values under both models, say $\hat y_{M_1}$ and $\hat y_{M_2}$. Now, we expect $\hat y_{M_1},\hat y_{M_2}$ to be correlated, so take $$ \hat{d} = \hat y_{M_1}-\hat y_{M_2}, $$

and apply a $t$-test or $z$-test to $\hat d$. If you reject the null then, the two fits have different population averages, i.e. the fits are different. Alternatively to $t$ and $z$, you may use non-parametric tests for differences in location for two paired samples.

If you want to test if the distributions of the fitted values under $M_1$ and $M_2$ are the same, you may try something like the Anderson-Darling test or the Kolmogorov-Smirnov test.

Improve this answer
$\endgroup$
2
  • $\begingroup$ Thanks for your answer! As far as I understood, BIC and AIC are for comparing which model fits best to my data. However I'm having only one model (the formula which I mention) which I know describes the data. I rather want to remove measurements with low quality data then find/compare alternative model which describes the data better. $\endgroup$
    – Manuel
    Sep 12, 2022 at 9:49
  • $\begingroup$ I don't know in your specific application but, usually we try to adapt model to the data and not adapt the data to the model. The reason is that, if you do the second way, you are likely to introduce selection bias. In practice this means that you will get a perfect model for that specific sample but a terrible one for the population; and it's the population you care about, not the sample. $\endgroup$
    – utobi
    Sep 14, 2022 at 8:59
1
$\begingroup$

I only have one (depending on the perspective maybe 2) measurements per timepoint

Having only one measurement per timepoint brings you in a difficult situation. This makes it difficult to analyze whether the discrepancy between your fit and the data points is due to noise or due to a wrong model.

You need some independent measure of the expected noise.

  • If you would have multiple measurements of the same situation (maybe you have at least a few cases with multiple measurements per time point?) then you can use the method described here: Linear regression: F-test for lack of fit (using ANOVA to test regression model) - intuition?

    A technical/practical problem when applying this to your model might be that the source of error might be misrepresented. In your case it might be that you have errors that are heteroskedastic, or measurements errors in the "independent" x-variable besides errors in the y-variable. Such complex cases will make the analysis more difficult.

  • Possibly you can have some theoretic considerations about the expected noise.

    For instance, if it is a reasonable assumption that the distribution is Poisson distributed (this assumption can be tricky because one might have over or under dispersion) then you can estimate an expected noise level based on the estimated mean. If your observed noise level is larger, then you do not have a good fit.

    In your case there might also have a priori models about the noise level. E.g. some physical models for growth dynamics and expected noise/variations in the growth curves.

Improve this answer
$\endgroup$
1
  • $\begingroup$ If you want goodness of fit in order to compare and select different models, then you might not need a model independent measure of the expected noise, and you can use the residuals to perform statistical tests. $\endgroup$
    – Sextus Empiricus
    Sep 9, 2022 at 12:48

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.