3
$\begingroup$

I have a $[k \times N]$ matrix of predictors / independent variables and a $[k \times N]$ matrix of predictands / dependent variables. I have uncertainty estimates for each predictor and each predictand. I use orthogonal distance regression (ODR, with scipys Python interface to ODRPACK) to estimate my predictands one column at the time (regressing $[k \times N]$ as independent and $[1 \times N]$ as dependent), but my estimate of β (estimated parameter values) suffers from strong collinearities between the input vectors, i.e.

$y_8 = 1.3 - 0.046x₁ + 1.45x₂ - 1.49x₃ + 2.2x₃ - 1.4x₄ + \ldots + 1.1x₈ + \ldots$

...which in turn leads to my model being poorly generalised to independent testing data. For physical reasons¹, I would expect that all but one factors of β are close to zero, and my initial guess for β is

$$ \beta_k = \cases{% 0 \quad k \neq i\\ 1 \quad k = i} $$

and I would expect the estimated $\beta$ to be close to it.

In practice, $k=12$ and $N$ is in the order of $10^3-10^5$.

How can I mitigate the problems caused by strong collinearities between the input vectors? Can I penalise large deviations from expected coefficients or otherwise regularise input data? Alternately, can I incorporate my errors-in-variables into methods better suited for collinear predictors, such as PLSR? I am looking for a practical solution that is reasonably straightforward to implement using Python and available libraries.


¹ Physically, I am comparing two N-channel radiometers with small, poorly known differences between corresponding channels, leading to small but nonzero expected differences in radiance. My regression aims to get a better understanding of the differences. Uncertainty estimates are essential to the project.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.