I'm using Partial Least Squares regression to relate two datasets: X has 40 variables, Y has 54 variables, both have 26 observations. Because I don't have a lot of observations, I'm using a permutation test to determine if the PLS components I get are significant or are overfitting: for each component, I'm comparing the singular values from my PLS to a permutation distribution built by shuffling the rows of X 1000 times (following this paper: https://pubmed.ncbi.nlm.nih.gov/31515054/).
I noticed, though, that the singular values I get in my actual data are significantly lower than those in the permutation distribution (p < 0.05 for the first two components, p < 0.01 for the third one). I've actually gotten a result like this a couple times now relating different data in my dataset with PLS.
How can I interpret a result like this? Does it imply something about the suitability of PLS for my data/something wrong about the procedure? Or (perhaps wishful thinking), might it suggest that there's a nonlinear relationship in the data which reduces the magnitude of the linear relationships found by PLS? Any help would be much appreciated!
More info about the dataset and procedures below:
X and Y had a bit of sparsity to each of them (20-30%), and were standardized before running the model. Most variables in each of them were normal-ish besides this, with a bit of skewness in some of the Y variables. I did a simulation with normally distributed random data of the same size and sparsity, and didn't find anything like the above issue. I'm using plsregress in MATLAB to estimate the components.