I am very new to regression as I have been reading "The Elements of Statistical Learning: Data Mining, Inference, and Prediction" by Hastie et al. on Standford's website this weekend.

My goal is to determine if I can make use of any techniques in my physics research. I am starting to get a slight feel for the LASSO technique, but wanted some input from someone that really knows the technique. It seems like from my reading regression really works best if you don't know your model a priori, which is half true in my case. I have two "systems" of the following:

  • System 1: I know the model equation. I have 4 complex valued equations of the form $H_j = \sum C_i X_i P(X) $ j=1..4;(where C are known constants different for each equation, X is my complex parameter space, and P are related to the Legendre polynomials). This is an infinite series in parameters, but I know physically that most terms above i=5 are basically 0. My experimental data then is known to fit 16 equations of the form $H_i*|H_j|^*$; hence the 16 equations are non-linear, and all correlated, and unfortunately currently under-determined (not enough data to get a unique solution).

For this system would LASSO be of any use, or would there be another regression technique to recommend? I currently just use a least squares approach.

  • System 2: This is more model based. My input is the hopefully correct output of system 1, my output is then some physical values. I don't know how many physical values I need, or what their values are in all cases. I believe this case really lends itself to LASSO, but not sure how to apply it. So here for this system, how would one apply it? The LASSO equation is: $\sum_i (y_i - \beta_0 - \sum_{j=1}^p x_{ij}\beta_j)$ under the constraint of the $\sum_j \beta_j$ being smaller than some user determined value.


  1. In my system how do I interpret the $y_i$, $\beta_j$, and $x_{ij}$ to be?
  2. Do I replace the least squares technique with this one, or does LASSO work on top of least squares, meaning, do I use both terms to get my best parameter set?
  3. Also, anyone know a good book that might give insight into how to program this into Fortran/C++ code?
  • $\begingroup$ glmnet is written in fortran $\endgroup$ – bdeonovic Feb 1 '16 at 23:14

Regarding your System 1:

Even when your model is known up to parameter values it may make sense to use LASSO or some other form of regularization (e.g. ridge regression or elastic net). For example, if you know that the true model is

$$ y = X \beta + \varepsilon, $$

but you don't know the values of $\beta$, the minimum mean squared error (MSE) estimator of $\beta$ will involve some shrinkage, while the OLS estimator will have no shrinkage and will generally yield a higher MSE. (I say "some shrinkage" because there is no analytical solution for an optimal amount of it. That might be part of the reason why we use OLS instead of a minimum MSE estimator.)

Regularization should be especially helpful in situations such as this:

This is an infinite series in parameters, but I know physically that most terms above i=5 are basically 0.

It will help sensibly tune down the effect of the high order members that are basically zero and the estimation variance of which may be boosting the mean squared error.

Regarding Question 2 (if I understand it correctly):

LASSO is a substitue for non-penalized regression (OLS). I don't think it makes sense to somehow run both sequentially, or however I should interpret the question "does LASSO work on top of least squares?".

  • $\begingroup$ I apologize that I don't understand the notation/terminology of regression. So let me see that I have it straight: In your equation, $y = X\beta + \epsilon$ I take to mean y is my measured data in system 1 which I called $y == H1 * |H3|^*$ for instance. $\beta$ is some unknown parameter set designed to force some parameters to 0, and X is my parameter set? $\endgroup$ – user2103050 Feb 1 '16 at 19:37
  • $\begingroup$ $y$ is a column vector containing the observations for the dependent variable (the regressand). $X$ is a matrix consisting of several columns where each column contains a vector of observations of a particular independent variable (also known as regressor or feature). $\beta$ is a parameter vector that you want to estimate. This is a very simple linear model, yours must be more complicated (nonlinear). But the idea should still apply: as long as you are estimating $\beta$ from the data it may make sense to apply some shrinkage. (I know my answer addresses only part of what you are after.) $\endgroup$ – Richard Hardy Feb 1 '16 at 19:43
  • $\begingroup$ Thanks, that explanation makes it much clearer what the terms signify. My equation is 1 independent variable (Angle) and the 16 (correlated through the parameters) dependent variables. So very unique when it comes to regression $\endgroup$ – user2103050 Feb 1 '16 at 21:07

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.