1
$\begingroup$

looking for some further help and to further my understanding of regression topic area.

So I am trying to compare regression models with two different data sets A and B.

Using dataset A, I fitted the regression model y= mx1+ nx2 + c this gave rqs =95%

I want to know how well this model works for dataset B. I have tried to understand this using two methods.

  1. Using the same variables x1 and x2, fit regression model for dataset B which gives me y=ox1+px2=c (different coefficients and intercept) with r sq = 75%

  2. I have used the regression eq obtained in original fit y= mx1+ nx2 + c and input the x1 and x2 from dataset B to obtain predicted values for y. I have then plotted the real y values against the predicted y values and assessed the r sq value.

Is any of these methods valid? Or is there a better method for comparing regression models? I have some initial limitations with using method 2 as the model is fitted the real values to a predicted value from a prediction. Can anyone help my understanding of this?

$\endgroup$
1
$\begingroup$

https://en.wikipedia.org/wiki/Chow_test discusses the CHOW TEST which can be used to test the hypothesis of a common set of parameters over k groups ( 2 in your case ). I routinely use this in a time series context to DETECT the point in time when the model parameters change significantly

$\endgroup$
  • $\begingroup$ Thanks for this, had a read through and looks like a useful test for future use! $\endgroup$ – user252775 Aug 5 '19 at 14:52
0
$\begingroup$

There is a logical gap between the two methods, if we want to be precise. The first method aims to assess whether the same variables significantly describe the data of dataset 2. While the second method assesses the out-of-sample performance of the equation estimated on the first dataset (I.e the ability to use the equation estimated in dataset 1 to predict the dependent variable in the dataset 2). Clearly there is a logical difference, because in the first method you are just looking at the common variables (with any coefficient), while in the second you are testing the feasibility of using the exact relationship estimated in dataset 1 to interpret dataset 2. So it depends on what your purpose is.. maybe the two datasets share the same relevant variables but with very different coefficients (in this case you may have the first method saying the variables are the same, because the same variables are significant in both the dataset, but their “true coefficients” vary significantly across the two datasets so the second method gives disappointing results because the first equation performs poorly in terms of fit in the second database)..

So if your objective is to predict dataset 2 based on the info in dataset 1 (as I believe reading your problem description), then choose the second method.. instead, if you want to assess whether the two common variables are significant in both the dataset (regardless the shape of the relationship and the coefficients), then choose the method 1 and test whether one/both the variables are significant for the model in dataset 1 and 2 (regardless the coefficients)

I also endorse the solution by @IrishStat (upvoted as such), to test whether the parameters of the relationship change across the two samples

$\endgroup$
  • $\begingroup$ Hi, Thank you for this explanation. I am indeed trying to assess how the model created using dataset 1, fits the data in dataset 2. Just to clarify, I am okay to use the decay of the rsq value in a direct comparison between the two equations. Eg. $\endgroup$ – user252775 Aug 5 '19 at 14:54
  • $\begingroup$ Yes, the "decay" is the exact term as i highlighted in my recent reply here stats.stackexchange.com/questions/420502/explained-variance/… notice also that, usually, the simpler the better, so if you have a simple need like testing the out of sample performance of an estimated model, just use simple and widespread methods to solve the problem (i.e. the decay of the $R^{2}$ in this case). $\endgroup$ – Fr1 Aug 5 '19 at 14:59
  • $\begingroup$ Hi, Thank you for this explanation. I am indeed trying to assess how the model created using dataset 1, fits the data in dataset 2. Just to clarify, am I okay to directly compare rsq values from these equations. Eg. equation 1 rsq is 95%, then when I plot the predicted values using this equation but data from dataset 2 against the real values in data set 2, I get an equation xreal= x_pred + c. I am concerned the different intercept means these two r sq are not comparable. Thanks again! $\endgroup$ – user252775 Aug 5 '19 at 15:00
  • $\begingroup$ Sorry posted first comment too early but looks like you replied before I had time to edit! Thank you! $\endgroup$ – user252775 Aug 5 '19 at 15:01
  • $\begingroup$ Wait.. if you want to compute the decay of the R2, just use the estimated equation in dataset 1 in order to compute the residuals of the same equation in dataset 2 (res_dataset2=y_dataset2 - mx1 - nx2 - c), and then do 1-Sum_squared_res_dataset2/Total-sum_squares_y_dataset2 .. and that will be fine $\endgroup$ – Fr1 Aug 5 '19 at 15:04

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.