Is there a way to do Coordinate descent but depending on the variable change the method applied to find the coefficient?

For example, apply a LASSO constraint to a predefined 3 variables and Ridge to the other 5? Or is this just elasticnet with a ratio parameter from 1 (meaning LASSO) to 0 (meaning Ridge) depending on the variable?

Main purpose is to be way more aggressive towards certain variables and shrink them to 0 while keeping others more intact.


I found this code to do LASSO with CD, could I just change the beta calculation in the loop or are there more complications with the CD algo?

def lasso_nb(X, y, alpha, tol=0.00001, maxiter=50000):
    n, p = X.shape
    beta = np.zeros(p)
    R = y.copy()
    norm_cols_X = (X ** 2).sum(axis=0)
    resids = []
    prev_mse = 10000
    for n_iter in range(maxiter):
        for ii in range(p):
            beta_ii = beta[ii]
            # Get current residual
            if beta_ii != 0.:
                R += X[:, ii] * beta_ii
            tmp = np.dot(X[:, ii], R)
            # Soft thresholding
            beta[ii] = fsign(tmp) * max(abs(tmp) - alpha, 0) / (.00001 + norm_cols_X[ii])
            if beta[ii] != 0.:
                R -= X[:, ii] * beta[ii]
        mse = np.mean((y - X @ beta)**2)
        if prev_mse - mse < tol:
            prev_mse = mse
    return beta, resids
  • $\begingroup$ It seems like you have already answered your question yourself. There's not much complications, since the cost function is convex, and coordinate descend should work without problems. $\endgroup$ Aug 14 at 20:22
  • $\begingroup$ @SextusEmpiricus great, didn't know if alternating would degrade the convergence. $\endgroup$
    – Tylerr
    Aug 14 at 20:24


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.