1
$\begingroup$

I have two discontinuous observational datasets that should roughly match up after linear scaling is applied to one and its offset is adjusted. They will not follow any kind of trendline, so techniques like linear regression may not minimize total variance.

For example, 1,2,3,2,1 and 102,104,106,104,102 have zero variance if the first set is scaled by 2 and then an offset of 100 is applied. My goal is to find the formula producing 2x+100 among two wandering real-world datasets.

Currently I linearly regress both sets and use the difference in x value and offset to rescale one of the sets to match the other, but I'm not sure if this is ideal as neither set is linear and goes up and down like a moving average.

$\endgroup$

1 Answer 1

1
$\begingroup$

You could consider your two datasets $D_1, D_2$ as samples from two different probability density functions (pdfs) $p_1, p_2$, resp. And then the question is how to transform $p_1$ to obtain some new pdf $f(p_1)$, where the transformation $f$ is only allowed to shift and scale, such that $f(p_1)$ is as similar as possible to $p_2$. Thus, for this to work, you have to:

  1. Turn your datasets into pdfs,
  2. find an appropriate similarity notion between pdfs, and
  3. find the $f$ that maximizes this similarity between $f(p_1)$ and $p_2$.

For (1): You can do this e.g. by creating some histogram of your data the integral of which is equal to one.

For (2): There are many distance notions for pdfs, and your domain knowledge should be used to pick the right one. I would recommend having a look at the Wasserstein distance. Once you have chosen the distance metric, you can use this as inverse similarity: the smaller the distance, the larger the similarity.

For (3): Just enter your data into a solver of your choice, e.g. optim, if you want to use R. You will have two parameters, the shift and the scale, that you can vary in this optimization. And the objective function is the Wasserstein distance between $p_1$ and $p_2$, which has to be minimized.

$\endgroup$
2
  • $\begingroup$ Is this made any simpler by both sets being time series with maximum one point per time division and the correlation not requiring any time shifting? $\endgroup$
    – Lucent
    May 23 at 5:16
  • $\begingroup$ @Lucent Could you please define what you mean by "total variance between two datasets"? $\endgroup$
    – frank
    May 26 at 6:26

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.