How should I perform a statistical comparison between the model fitting results on two separate datasets? The same model is applied to both sets, but the data are from two conditions, e.g. control and treatment.

I read that I need an F-test. But the formula for the F statistic (from wikipedia),

$$ F = \frac{\frac{RSS_1 - RSS_2}{p_2-p_1}}{\frac{RSS_2}{n-p_2-1}} $$ with $p$ as the number of parameters in a model and $RSS$ as the sum of residuals.

But in this formula, the models are the same, so $p_2 - p_1$ = 0, and then the formula isn't valid. So where am I going wrong?

Context:

  • My model equation is y = s-(s/((a+x)*b)),
  • I'm using the fitting procedure from lmfit.Model to find the parameters. (why lmfit? The rest of our processing is already in python, and I couldn't get the statsmodels package to handle general model specifications with parameters; but identifying the statistical method is the primary reason for asking)

EDIT: additional context as requested in comment

  • Overall aim: the dynamics of the network are for each node in the network to settle into a decision. The individuals can choose from one of two states, and we are interested in the conditions under which:
    • a) the fraction of nodes in network reaching the same state is maximised, and
    • b) how fast this happens.

Form of data: If we label the control experimental group #1 and the treatment group #2, the data looks like:

# time, average fraction of nodes with majority choice, experimental group

60,  0.52, 2
70,  0.53, 2
80,  0.55, 2
90,  0.54, 2
...

1200, 0.92, 2
  • The "average fractions to majority choice" are averages across the replicates carried out.
  • We have 15 replicates or more per treatment, and the analysis is on data already obtained.

  • group #1 has lower average fractions by the end of the experiment (because they don't have the same level of agreement in general).

  • The curve following a saturation equation (above) fits the data reasonably well and fits our hypotheses for the underlying dynamics. So we are not looking to select between models.

  • However, we do want to say whether the `treatment' makes a difference.

So, the question here is whether difference between the fitted curves is statistically significant. What is the most appropriate way to test this?

  • The provided context helps, but full context would help more. – Kodiologist Jan 26 at 19:58
  • Have you considered Root Mean Squared Error (RMSE)? It is a commonly used fit statistic that seems like it might be useful in this case. – James Phillips Jan 27 at 2:56
  • @JamesPhillips I can calculate this to check the curve fitting is ok, but I don't know how to solve my question with the values. How do you use RMSE in a statistical test between the different treatments? – Bonlenfum Jan 29 at 15:12
  • @Kodiologist I have added some more details on the research qu and the data. Please let me know if that helps you to a suggestion on what I'm trying to do! thanks – Bonlenfum Jan 29 at 15:15

It looks like what you really want to do is just to check (a) whether the mean time significantly differs between conditions and (b) whether the mean "average fraction of nodes with majority choice" significantly differs between conditions. Your model isn't necessary for either of these. For (a), you could use the Mann-Whitney $U$-test, or an independent-samples $t$-test on the logged time. For (b), you could use the Mann-Whitney again or Fisher's exact test (for the latter, you'll need to recode the fractions into node counts).

I still can't tell what the study is about, what the network means or comes from, how the stated model equation relates to the data, what the experimental manipulation is, etc., so it's possible I'm misunderstanding your situation.

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.