0
$\begingroup$

The data I have to analyze are as follows. There are three group of samples: the reference samples (REF, 100 samples) and two groups of contaminants (Ca, 100 samples and Cb, 10 samples). I have spectroscopic data (NMR data) for all the samples and the spectra within each group are very similar but their difference is definitely greater than experimental error. I want to train two PLS models to estimate the contamination of REF with Ca and Cb. I know that the NMR specrum of a contaminated sample is the linear combination of the specrum of the reference sample and of that of the contaminat, so I want to simulate the spectra of contaminated samples and use these simulated spectra to train the PLS model. So, my question is: which is a good set of simulated spectra to train the PLS models for Ca and Cb? Also, consider that I'm mostly interested with contaminations up to 10 %. For example, should I use one spectrum only to simulate a single contamination or can I use it to simulate many levels of contaminations? I have some concerns that using the same spectrum many times could bias the results of crossvalidation for determining the best number of PLS components.

$\endgroup$
  • $\begingroup$ I have prepared some real contaminated samples. Comparing their spectra with the simulated ones both visually and on the PCA. My judgement was that the difference is just noise. $\endgroup$ – user7669 Nov 27 '17 at 23:49
  • $\begingroup$ I cannot disclose the nature of samples, but to give an idea I can say that reference and contaminants are of very similar nature, like olive oil or wine from different brands/regions. $\endgroup$ – user7669 Nov 27 '17 at 23:57
1
$\begingroup$

I know that the NMR specrum of a contaminated sample is the linear combination of the specrum of the reference sample and of that of the contaminat

How do you know that? Can you really exclude the possibility of a new chemical species forming in the contaminated sample? Keep in mind: dealing with such chemical interactions one of the big points for PLS, and the species does not need to be a reaction product or even intermediate molecule, it can be something like surface species, dimers, conformation changes, ...

But PLS can only "learn" this if the spectra actually reflect the possibility of such interactions.

If you per se do not consider the possibility of interactions, you can decompose, say, on a set of PCs for REF plus contaminant spectrum (or a set of PCs of the contaminant if there's some known variability here as well). The PLS won't produce any further information if the training data didn't cover further variation.

At the very least I'd say that you need to validate your model with true contaminated samples. Otherwise you'll be creating a self-fulfilling prophecy which will almost certainly lead to a huge overoptimistic bias.


I have some concerns that using the same spectrum many times could bias the results of crossvalidation for determining the best number of PLS components.

That IMHO isn't the worst problem here as it can quite easily be prevented: when setting up your crossvalidation you can split so that training and testing are independent both reference and contaminant. I.e. you select particular reference and contaminant true spectra for testing, exclude any spectrum from training that either contains the reference in question or the contaminant spectrum in question.

But here's the catch: as you did not give any chemical justification why the linear combination of reference and contaminant spectra is more than the usual 1st approximation to the situation. So already before calculating the PLS we can say that you cannot possibly know whether your latent variables will be a suitable representation of the possibly more complex reality. It doesn't make sense to speak about findig the optimal number of latent variables in this case. Plus, the scheme you propose for producing the data means that we already know you will need at most one more latent variable than the number of principal components to describe REF. And here again, without knowing the lowest level of contaminant that is important to recognize, it is impossible to judge whether fewer components to describe REF would be sufficient. (Both for us and for your latent-variables optimization)


spectra within each group are very similar but their difference is definitely greater than experimental error

Without knowing far more in detail about your actual application/question it is impossible to say whether you can conclude anything useful from this.

You didn't even tell us what the contaminant spectra represent, e.g.: are they pure component spectra of 100 contaminants? Are they 100 spectra of contaminants isolated from real samples (with contamination)? Are they 100 spectra of contaminated samples? If so, are they representative for certain types of contamination that can occur/how were they selected?

$\endgroup$
  • $\begingroup$ Tanks for your very informative answer. Could you please add more detail on the PCA approach you suggested? If I do the PCA of the the REF spectra and the Ca or Cb spectra I see that the two groups are separated in PC1, but the dispersion of each group in PC1 is on the order of 10% of the separation between the groups. I was hoping that the PLS could deal with this by somehow forcing all the REFs to 0% and all the contaminants to 100%. I will check this by comparing the performance of the PCA and PLS models on real contaminated sample with known amount of contaminants. $\endgroup$ – user7669 Nov 27 '17 at 22:15
  • $\begingroup$ The PCA solution that came to my mind would be something similar to SIMCA (soft independent modeling of class analogis) of the REF class plus some additional information retrieval for the known contaminants - for contaminant a another class in the SIMCA may be feasible, but for contaminant b with only 10 spectra available you may have to revert to something more basic (e.g. spectra of the relevant species b (or also a) forms in REF). $\endgroup$ – cbeleites Nov 28 '17 at 12:02
  • $\begingroup$ "was hoping that the PLS could deal with this by somehow forcing all the REFs to 0% and all the contaminants to 100%": you can set up PLS to do this, yes. Whether it is sensible depends entirely on your particular situation. For food contamination scenarios such as you hint at it often is not sensible - but I cannot tell you more than these very general caveats without detail information. (Just in case: I'm not here to advertise, but I do offer chemometric consulting as professional service and then of course NDA or similar contractual agreements are avialable - so do other people) $\endgroup$ – cbeleites Nov 28 '17 at 12:04
  • $\begingroup$ Thanks. Maybe I will compare PLS with PCR. I will accept you answer as it explains how to do crossvalidation, which was my main concern. $\endgroup$ – user7669 Nov 29 '17 at 17:43

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.