2
$\begingroup$

I have two experimental datasets $D_1$, $D_2$. An algorithm $A$ is applied to both resulting in errors $E_1$, $E_2$. From $D_1$, $D_2$I can create another dataset by randomly sampling half of the data from $D_1$ and $D_2$ respectively and concatenate it to form $D_3$. Applying $A$ to $D_3$ yields an error $E_3$.

Am I allowed to compare the errors $E_1$ and $E_2$ against $E_3$ and from this conclude which dataset is best?

In more details: I am investigating 3 surface matching methods based on a deterministic surface matching algorithm $A$. Each surface matching method consists of the algorithm applied to a surface. Each method results in an error. There are 2 distinct surfaces, namely $S_1$, $S_2$ (I guess these are random variables since I am sampling the surfaces).

I have 10 realizations $D_1=s_{1,i}$, $D_2=s_{2,i}$, $i=\{1,…,10\}$ of the surfaces $S_1$, $S_2$ (from experiments). Each realization $s_{i,j}, i=\{1,…,10\}, j=\{1,2\}$, consists of 100 randomly sampled points from the real surface $S_j$ plus some noise. Hence, it is possible to calculate the errors $e_1,i=A(s_{1,i}), e_{2,i}=A(s_{2,i}), i=\{1,…,10\}$ and related statistics describing the method (mean, median, variance, related confidence intervals of the errors, …).

Getting towards my question: In my case I cannot create further realizations of $S_1$, $S_2$ nor is it possible to get independent realizations of the surface $S_3:=S_1 \cup S_2$. However, it is possible to simulate realizations of $S_3$ consisting of 100 points by randomly sampling equal portions from $s_{1,i}, s_{2,i}, i={1,…,10}$. It is possible to create lets say 1000 realizations $D_3=s_{3,i}, i={1,…,1000}$ of the surface $S_3$. From these it is possible to calculate errors $e_{3,i}=A(s_{3,i}), i={1,…,1000}$ and related statistics describing the method (mean, median, variance, related confidence intervals of the errors, …).

After doing this, is it valid to compare statistical values (e.g. mean, variance, ...) of $e_{1,i}, e_{2,i}, e_{3,i}$ to argue which method is best (I am asking since $D_3$ actually contains the same data as $D_1$, $D_2$)? What conditions have to be fulfilled for $D_1$, $D_2$ that it is valid to compare the results of $D_3$ to $D_1$, $D_2$?

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.