I am working on Approximate Bayesian Computation for a simple regression model. Currently, I am not sure how a distance metric in the setting of regression analysis should look like.
Imagine a simple linear regression and the corresponding scatterplot.
set.seed(123) x = runif(100,1,100) y = 2 + 3*x + rnorm(100,0,50) plot(x,y)
Assuming that the underlying data-generating model is a regression model with a specific intercept and slope, I would like to evaluate another set of datapoints (calles y') and assess how likely this new sample was generated by the same underlying model.
So, this (see below) new sample (y') should be judged as being similar to the one before (y) since it is generated by the same underlying model:
set.seed(456) y = 2 + 3*x + rnorm(100,0,50) plot(x,y,ylab="y'")
Note that the x-values stay the same, so the distance metric should only be concerned with the distance between the y-values (distance between y and y'). One way to judge the distance between two samples might be a correlation coefficient but unfortunately it is not an appropriate metric. The pure sum of squared distances between the y-values is not correct either. If you simply add to each y a small amount then the sum of squared distances between the old values (y) and the new ones (y') is small but the underlying model changes (the intercept is shifted by the amount added to the datapoints).
Is there any metric that is suitable for the scenario I described?
It it reasonable to draw any intercept and slope, 'add' this line into the first scatterplot and evaluate the distance as the sum of squared residuals. If this distance-measure is small, then the line that was chosen fits to the data well.
Although this approach is not wrong, I cannot estimate the residual variance $\sigma$ as a parameter (as it is estimated in maximum likelihood estimation procedures). If I'm correct, in order to be able to estimate $\sigma$ as well, I have to somehow judge the similarity between the actual datapoints y and y'.