0
$\begingroup$

Why does preprocessed test data change when calibration dataset (and model based on that data) changes? I have spectral, normalized datasets, the preprocessing was 1. derivative + autoscale.

For example: I have 2 datasets: 100 samples with 400 variables each as calibration data; 20 samples for test data. preprocessing as above, data was cross-validated as well. the model for PLS-DA was calculated using pls toolbox in matlab. when I plot preprocessed test data I get different results compared to situation when I used e.g. half of those calibration-set samples to create a model. and I can't plot preprocessed test data without calculating the model. how is it connected?

I also tried to put my 20-sample test data as calibration data just to create model, and the preprocessed data plot was also different from plots obtained before.

Improve this question
$\endgroup$

1 Answer 1

Reset to default
1
$\begingroup$

It has nothing to do with PLS-DA, it is related to autoscaling spesifically. While taking derivative (or smoothing) is applied per spectrum, the autoscaling does the following:

  1. Calculate the mean of each variable using all calibration set samples
  2. Subtract this mean from from each variable on both calibration and validation set
  3. Calculate standard deviation (STD) for of each variable using all calibration set samples
  4. Divide each variable by this STD on both calibration and validation set

Therefore, if your calibration set changes, the mean to be subtracted changes. Same happens with STD too.

Improve this answer
$\endgroup$
1
  • $\begingroup$ Now it's understantable for me, thank you. I have one more question to ask - is it reasonable to use the same preprocessing method as above but in opposite order (which is to use autoscale first and then 1. derivative)? $\endgroup$
    – Masquerr
    Commented May 3, 2019 at 19:16

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.