Multiple Linear Regression with more variables than samples

Question

I'm currently learning chemometrics for my work and I have a simple question about Multiple Linear Regression (MLR).

Just to explain the context: I am simply using UV-Vis-NIR spectra (2500 wavelengths) to quantify a molecule in presence of interfering species. I have built a calibration set which describes my concentration intervals in a complete and balanced way (50 samples), and a validation set that is basically real samples taken from a process (50 samples, independent from the calibration set). After some try/retry exercices and some optimizations with a chemometrics add-on for MATLAB, I've come up with a parsimonious PLS model (SIMPLS algorithm) that accurately predicts concentrations of the validation set. For now, because my validation samples are significantly different in concentrations and interfering species than my calibration set, I consider a model to be good if it correctly predicts my validation solutions: I do not use statistical tests such as t-tests.

However, after I tried MLR, I realized that the MLR model was significantly more robust with respect to interfering species (the root-mean-square error of prediction is twice smaller and some validation samples where the PLS model gives a prediction relatively far from reality are correctly predicted by MLR).

Here comes my question:

In almost every textbook or publication that I have read it is said that MLR is not applicable if we have more variables than samples, because the inverse of the $X'X$ matrix, where $X$ is the predictor block, doesn't exist. Yet, my MLR model is actually working better than my PLS model when, if I understand what I read correctly, the MLR shouldn't even work because I have more variables than samples (and my variables are supposedly very collinear).

Does the fact of having more variables than samples absolutely not prevent the model from being calculated, and therefore from making good predictions, but just makes the regression coefficients unstable and difficult to interpret? Or am I messing around, and having a well-functioning MLR model under these conditions should worry me about the relevance of my approach?

Thank you very much.

PS: I've learn (well, maybe unaccurately) the basics of chemometrics mainly with Tormod Naes and Harald Martens books, along with some publications. Do you have any book suggestion to pursue my learning? Ty again!

Answer 1

 In almost every textbook or publication that I have read it is said that MLR is not applicable if we have more variables than samples, because the inverse of the $\mathbf X′\mathbf X$ matrix, where $\mathbf X$ is the predictor block, doesn't exist.

You'll want to look up classical (aka ordinary) vs. inverse models here.

In chemometric (or chemistry), a classical calibration model follows the direction of the causality: A concentration $c$ causes some signal (here absorbance) $A$ and thus the would primarily predict spectra from concentrations. So this model for UV/VIS absorption is basically the Beer-Lambert-Law: $$\mathbf A = \mathbf C \mathbf S$$ with
$\mathbf A^{(n \times p)}$ the measured spectra with $p$ different wavelengths (variates, channels),
$\mathbf C^{(n \times m)}$ the concentrations of the $m$ different constituents aka components*, and
$\mathbf S^{(m \times p)}$ the pure component* spectra.

Estimating $\mathbf S$ needs only $n > m$ (of course, the more the better), and calculating the pseudoinverse is also possible.

In contrast, the inverse model is set up to directly predict concentration as function of signal (spectrum): $$\mathbf C = \mathbf A \mathbf B$$

here, the coefficients are $\mathbf B^{(p \times m)}$, and need $n > p$ in order to estimate.

---

There are a number of differences between the two models:

• The classical is efficient at estimating the calibration curve/pure component spectra, but (if available) the inverse model is more efficient at predicting concentrations.

• The classical model assumes the error to be on the signal, whereas the inverse assumes the error to be on the concentration.
  This has yielded some discussion whether the inverse models (such as PCR or PLSR) are really appropriate. But a point has been made that nowadays the instrument error on the signal is actually often lower than the error on reference analyses. (And I remember a slightly sarcastic comment that it is certainly lower than the usual preparation error in student labwork practica)

• The inverse needs many training samples, the classical needs all spectroscopically relevant constituents to be known. This includes the interferents. I.e., to properly set up a classical model here, you need reference interferent concentrations.

All in all, inverse calibration is typically preferred. Thus, if there are too few samples/cases $n$, some sort of regularization (such as PLS) is used.

---

Now, you describe the opposite: the PLS regression (which is a regularized inverse model) performing worse than the presumably classical linear model. This can happen: there is no guarantee that the PLS regularization is sufficienty good to do better than a classical linear model, in particular if you have very few samples compared to the number of wavelength channels (features, predictors).
It may thus happen that the unchecked cross-sensitivity of the classical model is less of a problem than instability and/or bias of the PLSR model.

Now, 2500 channels for a UV/VIS spectrum sounds like < 1 nm difference between adjacent channels. That's a lot compared to the usual band width in UV/VIS to be in accordance with the spectral resolution of your spectrograph and/or the spectral resolution needed for your application. One approach may be to reduce the number of spectral channels, e.g. by binning.
Also, if there are spectral regions that you know do not contribute information (neither about analyte nor about interferents), you may cut that manually. (Such external information is very powerful, good external knowledge can save you a whole lot of samples)

---

Literature

• search terms: classical/ordinary vs. inverse calibration, and forward vs. backward prediction yield you a buch of relevant papers.
• Textbook: Brereton's "Chemoetrics" (Wiley, 2018, 2nd ed.) discusses classical and inverse calibration (not sure about the 1st ed., but I'd expect it to be there as well). (a few pages)
• Handbook of Chemometrics and Qualimetrics Part B (Elsevier, 1998)
  (brief discussion)

---

* This can be chemical species, but also components to describe e.g. the baseline