How do I validate a regression model's inferences and not its predictions?

Question

Suppose over many years I collect data $X$ on a quantity of interest $y$ and some control variables $Z$. I fit an OLS model

$$y = \beta X + \delta Z + \epsilon $$ and use the coefficients $\beta$ and their statistical properties to understand how variables in $X$ relate to $y$. After a year I get new data $X^*$, $y^*$, and $Z^*$. I want to validate whether my inferences about the relationship were correct - how do I do that?

If I just refit the model using only new data and compare coefficients there is a big loss of power since I will have a lot less data to work with. On the other hand if I just tack on the new data at the end of the old and refit then it's biased as it still uses all the data the initial model was fit with.

Answer 1

As you desire to validate inferences, I suggest you validate the result of the beta vector and compare the respective covariance matrices. validate the result of the beta vector: (beta_1-beta_2)/sqrt(var(beta_1)+var(beta_2)). If you have normal data or a large sample, this test statistic will be approximately t-distributed with T-n (n number of parameters estimated) degrees of freedom. On a given significance level you can then reject or not reject your estimates of beta and hence, you can make statements whether the linear relationship that you want to find is time variant or not. Appart from that you could also analyse the RSS. I would suggest you calculate the RSS in a restricted setup (using old and new data at the same time) and a second time where you split the data into old and new and calculate RSS=RSS_old+RSS_new. Then calculate a test statistic based on that.