2
$\begingroup$

I am collecting 20 Mid-IR spectra from 10 samples by taking two measurements for each sample. I want to create a prediction model using PLS. What would be the best way to split the data between training and test sets for cross-validation, while avoiding overly optimistic results because of the replicates?

One idea is to take the average of each pair of replicate but that would leave me with only 10 samples.

Is there a way to create a relatively reliable model with the data I have and depending on the results, measure more samples and improve it?

$\endgroup$
  • $\begingroup$ Welcome to CrossValdated! Please have a look at the definition of the spectral-analysis tag - it is a topic totally different from spectroscopy! Spectroscopy questions may better be tagged as chemometrics. $\endgroup$ – cbeleites Jun 25 '17 at 13:31
1
$\begingroup$

First of all: unless the variance between your repeated measurements is negligible, I'd definitively recommend to keep both spectra. One of the advantages of PLS over PCR (principal component regression) is that it can downweight ("learn to disregard") sources of variance that are not correlated to the information you're after. But it can only do so, if both wanted and unwanted variance are covered by the training data.

Secondly, yes if you have repeated measurements or technical replicates (as opposed to newly prepared samples of the same concentrations), you should take this into account in cross validation and all measurements of the same sample should always be together in the same set (training or test).

A number of software tools nowadays allow you to specify such grouping, but even if that is not the case, wrapping a cross validation by your own that excludes those pairs of spectra isn't that difficult. For 10x2 spectra I'd say that it is even reasonable to do it manually.

Are you talking about a regression problem (quantitiation) or classification? For classification, you won't be able to get anything reasonable out of a total of 10 samples, sorry. (Even if you have 5 cases x 2 classes with spectra so different that you can tell by "naked eye" where each case belongs, you have at most 10 samples for testing (e.g. via cross validation) - leading to huge uncertainty on your test results.

For regression, your situation may be better (with a bit of luck). In that case, I'd suggest a leave-sample-out cross validation may be a suitable choice. If you decide to go for fewer models, it may make sense to think about particular non-random ways of splitting like Venetian blinds. Anyways, similar to what theGD suggests, I'd recommend to keep in mind that CV predictions of the outermost (e.g. in concentration) samples extrapolate the calibration range. It may be worth while to mark these predictions accordingly.

$\endgroup$
1
$\begingroup$

I believe taking average of your samples or not will not change your results significantly. A usual Mid-IR spectrum contains 2000+ variables (with the resolution of the instruments in my lab, at least) and modelling that much variable with 10 samples (even if the they have 2 spectra for each sample) does not sound good. The best option would be to increase the number of samples. Working with this small amount of samples is very very risky, you may end up with "good looking" model just by chance.

Second option is to assigning only few samples to the test set including their replicate measurements. If you put their replicate measurements to your original model you are likely to end up with an overfitted model, as you mentioned. Thus, if I have to, I would assign 3 samples (all 6 spectra, without averaging) of them to the validation set and the remaining 7 samples(their 14 spectra without averaging) to the calibration set and carry out the cross-validation step. After finding optimal number of components, test your model on validation set.

Also if you are modelling, say, concentration, then I would make sure that the samples having highest and lowest concentration are included in calibration set.

Edit: Also check this question where @cbeleites provides a paper and some intuition about the small sample size problem.

$\endgroup$
  • $\begingroup$ ... but please keep in mind that our paper is about classification (qualitative questions), not regression/quantitiation. $\endgroup$ – cbeleites Jun 25 '17 at 13:28
  • $\begingroup$ Hi @theGD, I have a question reagrding CV step for the calibration set. If save the calibration set also contains samples with replicates, in the CV step should I also take this into consideration and make sure the replicates are always in one set (train or test set for CV)? $\endgroup$ – Vickyyy Apr 15 at 21:13

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.