2
$\begingroup$

How does hypothesis testing work when a measurement is not a single number, but an entire spectrum?

For instance, suppose we want to distinguish a species of plant based on its absorption spectrum. By taking lots of measurements we obtain an estimate of the standard deviation of the population for species 1 and species 2 (represented here by the two different colours):

enter image description here

Note: in this particular case, there is an area with no overlap between the standard deviation bands (i.e. the shaded areas). Often this might not be the case. But there are always areas where there is not a complete overlap.

So, when a new measurement is taken, how do we decide whether the measured plant belongs to species 1 or species 2?

In statistics how do we generally deal with spectra as opposed to numbers?

$\endgroup$
3
  • $\begingroup$ Your final question is not a hypothesis test: it is a classification problem. Which kind of problem, then, are you trying to ask about? $\endgroup$
    – whuber
    Sep 19, 2018 at 17:27
  • $\begingroup$ But doesn't the classification of a measurement involve hypothesis testing? Each time a new measurement X is taken we have four hypotheses and each with a certain probability associated with it: (1) X is species 1 (2) X is not species 1 (3) X is species 2 (4) X is not species 2. I want to know how to correctly deal with this when X is a spectrum. If the problem can be best handled in a different way, I would appreciate some explanation. $\endgroup$
    – Merin
    Sep 20, 2018 at 6:43
  • 2
    $\begingroup$ You are using "hypothesis" in a non-statistical sense. It is not the sense of statistical hypothesis tests. A statistical hypothesis is a particular kind of subset of a family of distributions. Your problem sounds very much like classifying random objects. $\endgroup$
    – whuber
    Sep 20, 2018 at 13:48

2 Answers 2

2
$\begingroup$

An alternative to Romain's answer is to not look at the spectrum as a vector of size $n$ (to me that seems to give too much problems in finding out how the errors will be correlated and how to deal with those correlated errors to compute probability for a class).

Instead, you could use some dimension reduction technique and use the result of that to define some lower dimensional feature space. In that space you could perform more easily classification. (indirectly the dimension reduction computes the correlation for errors in the larger space)

This method makes sense because you can think of your spectrum as being the linear sum of several features, like presence of certain components, molecules, bonds, or whatever creates the spectrum.

PCA is used a lot in analyzing spectra.

https://scholar.google.com/scholar?q=pca+spectrum+classification

$\endgroup$
0
2
$\begingroup$

It seems like your question relates to a classification problem, more than hypothesis testing, even though it could be thought of that way.

You have to consider your spectrum as a $n$-dimension vector ($n$ being either the number of points defining your spectrum, or a discrete sampling of your data if it's continuous).

In that view, what you show on the plot looks like 2 clusters (red and blue), represented by 2 $n$-dimension cluster centers (as vectors); and confidence intervals on whether the spectra should lie if they belong to each family.

Then, you have several ways of dealing with your problem. You could for instance assume that your families are gaussian processes, and given a new spectrum, compute the $n$-dimension vectors of probabilities that each point was generated by a cluster (for both clusters). This would give you a vector of probabilities, and the one with the highest norm would tell you what cluster to put your new spectrum in.

In a more general approach, you could compute the distance between your new vector and each cluster center, according to whatever distance is relevant to your problem (euclidian distance, earth mover's distance...).

If you're trying to determine a "score" function of the fact that a new spectrum was generated by either class, you can use the likelihood approach, by computing $L(\theta|x) = p(x|\theta)$ where $\theta$ represents the distribution itself (shape of distribution at each wavelength), and the set of hyperparameters for this distribution (for instance, means and variances). This is actually about the same as my first proposal, presented in a more formal way.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.