I have a $[k \times N]$ matrix of predictors / independent variables and a $[k \times N]$ matrix of predictands / dependent variables. I have uncertainty estimates for each predictor and each predictand. I use orthogonal distance regression (ODR, with scipys Python interface to ODRPACK) to estimate my predictands one column at the time (regressing $[k \times N]$ as independent and $[1 \times N]$ as dependent), but my estimate of β (estimated parameter values) suffers from strong collinearities between the input vectors, i.e.

$y_8 = 1.3 - 0.046x₁ + 1.45x₂ - 1.49x₃ + 2.2x₃ - 1.4x₄ + \ldots + 1.1x₈ + \ldots$

...which in turn leads to my model being poorly generalised to independent testing data. For physical reasons¹, I would expect that all but one factors of β are close to zero, and my initial guess for β is

$$ \beta_k = \cases{% 0 \quad k \neq i\\ 1 \quad k = i} $$

and I would expect the estimated $\beta$ to be close to it.

In practice, $k=12$ and $N$ is in the order of $10^3-10^5$.

How can I mitigate the problems caused by strong collinearities between the input vectors? Can I penalise large deviations from expected coefficients or otherwise regularise input data? Alternately, can I incorporate my errors-in-variables into methods better suited for collinear predictors, such as PLSR? I am looking for a practical solution that is reasonably straightforward to implement using Python and available libraries.

¹ Physically, I am comparing two N-channel radiometers with small, poorly known differences between corresponding channels, leading to small but nonzero expected differences in radiance. My regression aims to get a better understanding of the differences. Uncertainty estimates are essential to the project.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.