Application of Classical Multidimensional Scaling in Matlab on new data

Question

I would be happy if someone gives me a hint for two following questions in Machine Learning - related application of Classical Multidimensional Scaling in Matlab:

1.) It is recommended (but in my experience rarely done) to apply feature extraction (and hence the feature projection methods as Classical Multidimensional or PCA ) separately on the training, validation and test set to get a more realistic model performance estimation during the model validation. Furthermore, I would get the same question if I would apply my models after model deployment in a real environment with new data. How can I use the classical multidimensional scaling separately on independent data sets after I have trained my ML model? Do I firstly project my training data on the lower dimension and later provide this determined lower dimensionality to the Classical multidimensional scaling of validation/test/new data?

2.) Is the workflow right: data normalization --> classical multidimensional scaling --> again data normalization on the reduced data? I have provided an example code. I am appreciating every hint! Thank you very much!

Regards, Denys

%% Using of Hold-Out Validation with training, validation and test-set

%% Classical Multidimensional Scaling Training data
DataTrain = normalize(DataTrain); % normalization
D = pdist(DataTrain,'euclidean'); % pair-wise distances
[Y,e] = cmdscale(D); % carry out classical multidimensional scaling
CumSumEig = cumsum(e./sum(e)); % cumulated sum in order to identify the cumlated relevance
LowDim = find(CumSumEig > 0.95, 1, "first"); % dimensions with cumulated relevance over 95 % 
DataTrain = Y(:,1:LowDim); % reduced data

%% Validation 
% How to reduce the dimensionality of the validation data, because it is
% recommended to apply feature extraction (and hence the feature projection
% methods) separately on the training, validation and test set?
DataValidation = normalize(DataValidation); % normalization
D = pdist(DataValidation,'euclidean'); % pair-wise distances
[Y,e] = cmdscale(D); % carry out classical multidimensional scaling
DataValidation = Y(:,1:LowDim); % reduced data

%% or application to new data
DataNew = normalize(DataNew); % normalization
D = pdist(NewData,'euclidean'); % pair-wise distances
[Y,e] = cmdscale(D); % carry out classical multidimensional scaling
DataNew = Y(:,1:LowDim); % reduced data
% Again Data normalization before making predictions?
DataTrain = normalize(DataTrain);
DataNew = normalize(DataNew);

Additional information after some research (on 19.11.22): Taking in account the comments so far I agree that the application of Classical MDS on out-of-sample data is probably not a recommended approach. nevertheless I found after some research that my question might be answered in the following paper: "The out-of-sample problem for classical multidimensional scaling" (doi:10.1016/j.csda.2008.02.031).

Answer 1

If you do MDS on two different data sets, you get different subspaces of your original data space. If you train a ML model on one subspace, there is no reason to believe that same model would even work if applied on data taken in a different subspace, this is particularly evident if these subspaces can have different number of dimesion, but even if you fix them to have the same dimensionality (like you do) these dimension could be describing different axes or differentiating factors in the original data, and a model trained on a subspace could become meaningless in another different subspace.

Same goes for normalization: let's say you normalise your test data and you fit a forest/tree on it. You also normalise your test set, but this set of data has different scales and centers, so a data point that goes to 0 in the training set, would become something more or something less in the test set, depending on the mean and variances of this set. As a result, the tree or forest could route this data point to different leaves than it would have otherwise and result in a wrong prediction.

So, what you should do is this:

• you normalize your training data, and then you use the same scaling parameters to rescale validation/test data (don't normalise them again from scratch)
• you use MDS to learn a subspace from training data, then you use the same learned subspace to fit new data. I don't know how to do this on matlab because I never worked in matlab, but it's definitively possible with some other packages. Don't just save the number of dimensions from the training set, save the whole manifold, otherwise you could be learning different subspaces from each data set (that's particularly likely with MDS which is non-deterministic).
• You train a model on your training set, and then you use that learned model to make predictions on new data.

Also follow the recommendations Christian gave you on the comments, those are both good advice.