I have a situation where I have two samples : Sample A and Sample B of pairs of data $(x,y)$ that I do now know if they belong to the same population. I perform a regression on Sample A and another regression on Sample B, such that $\hat{y}=f_1(x;\theta_1)$ and $\hat{y}=f_2(x;\theta_2)$. After performing regression, I inspect their fits and both curves look quite similar given that $f$ is parametrized by $\theta$. In other words, $\theta_1\sim\theta_2$, or $MSE(\theta_1,\theta_2)<\epsilon$, or any distance metric between $\theta_1$ and $\theta_2$ is `small'.

However: How do I statistically prove that both samples belong to the same population, and that the curves are the same, and that the difference in curve fitting are a product of sampling error and noise? Is there a way to provide confidence intervals or bounds to my assertion? Is this related to gaussian processes? Can I do a multiple rounds of t-testing on each of the parameters in $\theta$ after bootstrapping data points across both samples to get an equivalent of error bars?

Similar questions, but applied to time-series (so perhaps it is still applicable to my problem), although my problem is not a time-series one. In addition, the samples I am collecting data from are not fully overlapping in input $x$: i.e. suppose sample A is sampled from $[0,x^*]$, and sample B is sampled from $[x^*, x']$, s.t. $0\leq x^* \leq x'$:

Significant difference between time series - Can I do this?

test if two linear fits are different

  • 1
    $\begingroup$ If the fitted curves are effectively the same, then you should be able to run each data set through the other data set's fitted curve and have the same residual error statistics such as RMSE and R-squared from its own fitted curve. $\endgroup$ – James Phillips Aug 24 '18 at 1:26

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.