Find a linear scale factor and offset that minimizes total variance between two observed data sets I have two discontinuous observational datasets that should roughly match up after linear scaling is applied to one and its offset is adjusted. They will not follow any kind of trendline, so techniques like linear regression may not minimize total variance.
For example, 1,2,3,2,1 and 102,104,106,104,102 have zero variance if the first set is scaled by 2 and then an offset of 100 is applied. My goal is to find the formula producing 2x+100 among two wandering real-world datasets.
Currently I linearly regress both sets and use the difference in x value and offset to rescale one of the sets to match the other, but I'm not sure if this is ideal as neither set is linear and goes up and down like a moving average.
 A: You could consider your two datasets $D_1, D_2$ as samples from two different probability density functions (pdfs) $p_1, p_2$, resp. And then the question is how to transform $p_1$ to obtain some new pdf $f(p_1)$, where the transformation $f$ is only allowed to shift and scale, such that $f(p_1)$ is as similar as possible to $p_2$. Thus, for this to work, you have to:

*

*Turn your datasets into pdfs,

*find an appropriate similarity notion between pdfs, and

*find the $f$ that maximizes this similarity between $f(p_1)$ and $p_2$.

For (1): You can do this e.g. by creating some histogram of your data the integral of which is equal to one.
For (2): There are many distance notions for pdfs, and your domain knowledge should be used to pick the right one. I would recommend having a look at the Wasserstein distance. Once you have chosen the distance metric, you can use this as inverse similarity: the smaller the distance, the larger the similarity.
For (3): Just enter your data into a solver of your choice, e.g. optim, if you want to use R. You will have two parameters, the shift and the scale, that you can vary in this optimization. And the objective function is the Wasserstein distance between $p_1$ and $p_2$, which has to be minimized.
