I have a system that simulates some physical phenomenon. The true behavior is assumed to be described with a linear function $x(t) = a t + b$. Due to the randomized nature of the simulation each run produces different values for $x$.

Now I have several runs sampled at evenly spaced time points $t_i$. I would like to check a hypothesis that all the runs simulate the same behavior up to some random fluctuations, that is $y^k_i = a t_i + b + \varepsilon_i^k$, where $k$ is the number of the run and $\varepsilon$ — some random error.

From now I see two possibilities of such testing:

  1. Fit every sequence $(t_i, y^k_i)$ with a linear trend obtaining individual $a_k, b_k$ and somehow compare $a_k, b_k$ with fit's RSS. This sounds promising.
  2. Fit all the data with single fit, subtract it from all the data and perform testing on the residuals. The idea is that this single fit should be close to the true behavior (under the null hypothesis) and the residuals may be compared as independent values. This obviously won't work if the runs are very different.

My question: is this problem (or similar) known and studied in the literature? I appreciate any references.

UPD. The original question was unclear, so I'd add some real life example. Consider we have two groups of people (males and females). For each group we have meashured the height $h$ and the shoe size $s$ of each member. From the observations we deduce that shoe size changes linearly with height, i.e. $s \approx a h + b$. How can we now check whether this linear law is same or different for men and women?

  • $\begingroup$ Is there some reason why you can’t answer this question based on the parameter settings for each of the simulations? $\endgroup$
    – EdM
    Commented Sep 7, 2019 at 20:39
  • $\begingroup$ The parameters of the simulation itself are fixed. The idea is to check whether simulation results are reproducible or just random $\endgroup$
    – uranix
    Commented Sep 8, 2019 at 8:07
  • $\begingroup$ This seems like you are just testing whether your randomization worked. $\endgroup$
    – Peter Flom
    Commented Sep 8, 2019 at 12:38

2 Answers 2


It seems like a perfect solution for your problem can be found via the lack-of-fit test. The test requires that more than one observation is available for some time points $t_i$, which seems to be the case for your problem since (I believe) each run $k$ has an observation at time point $t_i$. The null hypothesis is that $$H_0: \exists \, a, b \in \mathbb{R}\,\mathrm{ such \,that } \, \mathbb{E}[y_i^k] = a t_i + b \, \forall i, k,$$ and iid normality of the errors is assumed.


In metalearning we have something called "complexity measures", it estimates the complexity of a dataset and for metalearning dataset with the same complexity will have equal biases so you should apply the same space hypothesis to them.

This article is a good reference on complexity measures, for regression: Data complexity meta-features for regression problems

For classification: How Complex is your classification problem? A survey on measuring classification complexity

This book presents general metalearning: Metalearning: Applications to Data Mining (Amazon)


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.