3
$\begingroup$

I have about one hundred groups of chemical testing results data. Within each group there are between 1 and 200 chemical testing results. Each set of results contains testing data for 5 chemicals. Within each group the testing results may be from the same original sample or they may have come from a completely different sample. My task is to try and determine which testing results could have come from the same sample.

So an example from one group looks something like this: 

| Test | Chem1 | Chem2 | Chem3 | Chem4 | Chem5 |
| A    | 0.01  | 0.01  | 0.51  | 0.09  | 0.42  |
| B    | 0.09  | 0.01  | 0.51  | 0.09  | 0.42  |
| C    | 2.66  | 0.01  | 0.51  | 0.09  | 0.42  |
| D    | 2.56  | 0.01  | 0.51  | 0.09  | 0.42  |

The first approach I took is not working. I tried taking the Euclidean distance. This does not work because in the above example A and B are closer together than C and D. But A and B can not be from the same sample as B is 9x A and this should not be possible given the precision of the test. I tried scaling the data but it does not address this problem.

What I think I am looking for is a distance/similarity metric that measures the ratio between the two rows of data. Is there a standard metric I can apply in this case?

Or is there an entirely different approach I should consider?

Improve this question
$\endgroup$
3
  • $\begingroup$ This seems to depend on whether you should think of the differences as multiplicative or additive. $\endgroup$
    – gung - Reinstate Monica
    Aug 10, 2015 at 16:23
  • $\begingroup$ Can you explain a bit more about what you mean... Maybe you can help me answer your question. If the true concentration of one of the chemicals in the original sample was 1, one would expect all of the results to be withing 15% or so of 1. This would be the case for all of the chemicals. $\endgroup$
    – Ian Wesley
    Aug 10, 2015 at 16:44
  • $\begingroup$ I am considering just brute forcing it and looping through to check if each chemical's testing results are within 15%. But I figured there is distance/similarity metric that would get me close to the same results. Or maybe there is another approach? $\endgroup$
    – Ian Wesley
    Aug 10, 2015 at 16:49

4 Answers 4

Reset to default
2
$\begingroup$

I agree with @Anony-Mousse. Using anything in statistics, simply because it is the default will never be the best way to go. Take your situation as an example: At the core of Euclidean distance is the difference between two values.
$$ D_E(x_i,\ x_{i'}) = \sqrt{\sum_j (x_{ij}-x_{i'j})^2} $$ That means Euclidean distance assumes the relationships are fundamentally additive in nature. Using your data:
\begin{align} &.09-.01 &<& &2.66-2.56 \\ &\qquad\ \ .08 &<& &.10 \end{align} Your comment that "B is 9x A" implies that the theoretical background in chemistry suggests the relationship should be multiplicative instead:
\begin{align} &\frac{.09}{.01} &>& &\frac{2.66}{2.56} \\[5 pt] &\quad 9 &>& &1.04 \end{align} So using ratios instead of differences will be appropriate here. Using ratios directly is not recommended, though, because ratios $<1$ scale differently than ratios $>1$ (i.e., $1/2 = .5$, but $2/1 = 2$). We can make the ratios symmetrical by taking the log of the ratios, but taking the log of a ratio is mathematically equivalent to taking the difference of two logs. Thus, if you preprocess your data by taking the log of every variable, you should be able to use Euclidean distance on the transformed data.

If I understand your situation correctly, this will give you the results you expect.

Improve this answer
$\endgroup$
3
  • $\begingroup$ Taking logs before analysis also makes sense from the subject matter, as in my experience errors in chemical assays are typically proportional to the observed values. In the log scale such proportional errors will become relatively independent of observed values, better meeting the assumptions of many statistical tests. $\endgroup$
    – EdM
    Aug 13, 2015 at 12:31
  • $\begingroup$ What are the go-to methods to do this when values can be negative too? $\endgroup$
    – thorwhalen
    Dec 22, 2021 at 21:06
  • $\begingroup$ @thorwhalen, from my answer I would argue that using anything in statistics, simply because it is the go-to method will never be the best way to go. You have to think about what makes sense in your situation. What does a ratio of a negative to positive value mean in your case? Once you have figured that out, everything else should flow easily enough. You may want to ask a new question, providing enough detail about your situation for people to be able to help. $\endgroup$
    – gung - Reinstate Monica
    Dec 22, 2021 at 21:40
1
$\begingroup$

Don't approach this from math/statistics.

Approach this from chemistry.

What constitutes a similar sample from a chemistry point of view?

Once you've solved the chemical problem, then you can try to map this into mathematics. Either by using a customized distance, or by preprocessing the data into a format sich that e.g. Euclidean distance captures your intentions.

Improve this answer
$\endgroup$
0
$\begingroup$

Yes there is a standard measure. It is called the Pearson correlation coefficient. It is available in scientific software such as R and scipy but it is also straightforward to compute yourself.

Improve this answer
$\endgroup$
0
$\begingroup$

From

Cha, Sung-Hyuk. "Comprehensive survey on distance/similarity measures between probability density functions." City 1.2 (2007): 1.

enter image description here

You can read the reference for a technical explanation, but, intuitively, some things to point out:

  • You (probably) want to be using an actual distance metric (https://en.wikipedia.org/wiki/Metric_(mathematics))
  • You need a way to handle 0 values or 0 differences, which could get dangerous if you just start dividing things based on an intuition
  • As-shown in the picture, these won't actually work when both values are 0, but you can just define that the distance between two 0 values is 0
  • Squaring the differences will give more weight to larger ones (compared to taking the absolute value of the differences)
Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.