2
$\begingroup$

Is there a way to do Coordinate descent but depending on the variable change the method applied to find the coefficient?

For example, apply a LASSO constraint to a predefined 3 variables and Ridge to the other 5? Or is this just elasticnet with a ratio parameter from 1 (meaning LASSO) to 0 (meaning Ridge) depending on the variable?

Main purpose is to be way more aggressive towards certain variables and shrink them to 0 while keeping others more intact.

edit

I found this code to do LASSO with CD, could I just change the beta calculation in the loop or are there more complications with the CD algo?

def lasso_nb(X, y, alpha, tol=0.00001, maxiter=50000):
    n, p = X.shape
    beta = np.zeros(p)
    R = y.copy()
    norm_cols_X = (X ** 2).sum(axis=0)
    resids = []
    prev_mse = 10000
    for n_iter in range(maxiter):
        for ii in range(p):
            beta_ii = beta[ii]
            # Get current residual
            if beta_ii != 0.:
                R += X[:, ii] * beta_ii
            tmp = np.dot(X[:, ii], R)
            # Soft thresholding
            beta[ii] = fsign(tmp) * max(abs(tmp) - alpha, 0) / (.00001 + norm_cols_X[ii])
            if beta[ii] != 0.:
                R -= X[:, ii] * beta[ii]
        mse = np.mean((y - X @ beta)**2)
        resids.append(mse)
        if prev_mse - mse < tol:
            break
        else:
            prev_mse = mse
    return beta, resids
Improve this question
$\endgroup$
2
  • $\begingroup$ It seems like you have already answered your question yourself. There's not much complications, since the cost function is convex, and coordinate descend should work without problems. $\endgroup$
    – Sextus Empiricus
    Aug 14 at 20:22
  • $\begingroup$ @SextusEmpiricus great, didn't know if alternating would degrade the convergence. $\endgroup$
    – Tylerr
    Aug 14 at 20:24

0

Reset to default

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.