# Model selection based on the closest fit to another model

For each subject I have two measures, let's say $$X$$ and $$Y$$, that correspond to the same measurement but each from a different measurement device. In theory we assume that ideally these two measurements should be identical, however, due to the different measurement devices we observe some differences between (hopefully systematic).

1. I'm predicting $$X$$ based on other predictor, let's say age: $$X \sim b_0 + b_1age$$. I do the same for predicting $$Y$$ based on the age: $$Y \sim b_0 + b_1age$$ The two models of course end up being different - we observe bias between.

2. I randomly select half of the measurements from $$X$$ and the remaining half of the measurements from $$Y$$ (subjects do not overlap). Basically, I'm mixing these measurements from two devices where half of the measurements are from one device and the other half from another and let's denote each measurement in this sample as $$Z$$. Now I make two linear models:

1) $$Z \sim b_0 + b_1*age$$ (I do not account for differences between measurement devices)

2) $$Z \sim b_0 + b_1*age + b_2*device$$ (I account for possible differences between measurements devices)

My goal is to figure out which model on average (1) or (2) after running step 2 many times would be closer to the $$Y \sim b_0 + b_1age$$ model which is basically the ground truth fit I would get if all the measurements were from the same device. In short, I'm trying to figure out whether including device as a covariate on average would make the fit from a mixture closer to the actual fit $$Y \sim b_0 + b_1age$$.

The reason I'm doing is that usually part of the data is acquired using one measurement device and another part with another device. However, here I have a sample dataset for which I know measurements from both devices and I can experiment mixing measurements from both devices for separate subjects.

The question is: how could I compare (1) and (2) models to the $$Y \sim b_0 + b_1age$$ model? For now I compare percent absolute differences between fitted values of $$Y \sim b_0 + b_1age$$ and (1) AND between $$Y \sim b_0 + b_1age$$ and (2).

Not sure if I fully understood the question, but based on my reading you seem to be looking for a way to test if device has a systematic effect. Your proposed way of doing it seems to be assuming that devices only differ in their mean bias.

So, since your models are nested (one of them contains all the terms of another), you can use an F test to check if the addition of separate means for your devices, the $$b_2 device$$ part, explains more of remaining variability in the data compared with not adding such term.

In this setting $$Z \sim b_0 + b_1age$$ would be the null model and $$Z \sim b_0 + b_1age + b_2device$$ the alternative. In other words, the null has $$b_2=0$$.

If by chance you are working with R:

fit0 <- lm(Z ~ age)
fit1 <- lm(Z ~ age + device)
anova(fit0, fit1)


A significant result from this procedure would imply that you can do a better job of predicting $$Z$$ when you have information about both: age and device.