Pattern of out-of-sample reconstruction error in NMF cross-validation: why is it monotonically decreasing? [duplicate]

Question

I am using nonnegative matrix factorization, NMF (in its variant OPNMF, which is subject to additional orthogonality and $H = W^TV$ constraints) to factorize a dataset.

To find the optimal number of components in my data, I am doing split-half cross-validation. This means, I split my original dataset in two random splits for some thousands of times, and at each iteration I run opnmf on both splits for a given number of components K. Then I reconstruct the split matrices, in-sample and out-of-sample: $\widehat{V_1,_1} = W_1*W_1^TV_1$ or $\widehat{V_1,_2} = W_2*W_2^TV_1$ and I calculate the reconstruction error by simple difference: $insampleRecErr = \widehat{V_1,_1} -V_1$ and $outofsampleRecErr = \widehat{V_1,_2} -V_1$

I do this for a large number of components, going up to almost the entire number of dimensions in my data.

However, the out-of-sample reconstruction error is not behaving as I expected. I was expecting the in-sample and out-of-sample reconstruction errors to follow the classic trend of being (1) monotonically decreasing for in-sample reconstruction, and (2) for out-of-sample reconstruction, to decrease until the optimum number of parameters, and then increase again when the model overfits.

What I see, instead, is that also the out-of-sample reconstruction error decreases monotonically at increasing number of components:

I tried creating synthetic datasets suited for testing NMF algorithms (via the synthethicNMF function in an R package), but I observe the same behaviour also in these artificial data. This seems to suggest that it is related to NMF per se, and not my data.

Can anyone help me understand why is out-of-sample recontruction error also decreasing motonically? In other sources (here), this is not observed. So, it doesn't seem like it is a property of NMF to achieve better reconstruction at increasing number of components. But if so, what would be a better way to assess generalization for NMF?

Thank you very much in advance!

Answer 1

not sure if it's correct to do so but I found the answer to my question so I'll answer myself:

the way I am (was) addressing the issue is not correct.

In fact, I was using NMF to factorize a matrix, and then trying to reconstruct the matrix using its own values (both when the dictionary was derived from the same matrix, of from another dataset).

This is NOT correct, as explained here about PCA: How to perform cross-validation for PCA to determine the number of principal components?

More correct approaches to perform CV in these contexts are provided in this article: Cross-validation of component models (Bro et al., 2008), and an alternative method for performing CV in SVD and NMF has been suggested in: Bi-cross-validation for SVD and NMF (Owen and Perry, 2009).

Hope it might help someone else!