Why I am getting bad result for the hypertuning parameters I am trying to learn machine learning with 3 very common datasets (Boston Housing Price, Red Wine Quality, and Bike). I am using 3 ML models (LinReg, Lasso, and Ridge). The LinReg is the default of the Scikit-learn but for Lasso and Ridge, I am doing the hyper tuning.
I have 3 accuracy metrics (MAE, MSE, R2). The overall accuracy is given below
Dataset     Model       MAE         MSE         R2
House       LinReg      2.96        19.60       0.74
House       Lasso       4.58        47.44       0.39
House       Ridge       5.39        65.25       0.16

Dataset     Model       MAE         MSE         R2
Wine        LinReg      0.50        0.42        0.35
Wine        Lasso       0.68        0.65        -0.0008   
Wine        Ridge       0.66        0.62        0.05

Dataset     Model       MAE         MSE             R2
Bike        LinReg      106.56      20812.63        0.37
Bike        Lasso       118.21      25732.92        0.23
Bike        Ridge       136.31      31186.12        0.07    

You can see, the MAE and MSE are always high for the Lasso and Ridge than the LinReg.  Even, I am getting negative R2 for the Wine dataset and Lasso model.
I did very basic data pre-processing and mostly not any feature engineering. As they are very basic datasets.
I am using Optuna for hypertuning. At the time of hypertuning, I am doing 5 fold cross-validation, neg_mean_squared_error as scoring function and set direction="minimize".
For the Lasso and Ridge T am tuning only alpha and the ranges in between 1e-7, 1e2.
def lasso_objective(X_train, y_train, kfolds) -> Callable[[Trial], float]:
    def objective(trial: Trial) -> float:
        args: Dict = dict(
            alpha=trial.suggest_float("alpha", 1e-7, 1e2), 
        )
        estimator = Lasso(**args)
        scores = cross_validate(
            estimator, X=X_train, y=y_train, scoring="neg_mean_squared_error", cv=cv_method, n_jobs=-1
        )
        return float(np.mean(scores["test_score"]))

    return objective


def tune(objective):
    study = optuna.create_study(direction="minimize")
    study.optimize(objective, n_trials=200)
    params = study.best_params
    best_score = study.best_value
    return params

I am supposed to get better results for the Lasso and Ridge (as I am doing tuning here). Could you tell me why I am getting bad MAE and MSE for them? What should I do to get better results for them?
 A: TL;DR -- The code in the question maximizes model misfit because it uses "minimize" in conjunction with neg_mean_squared_error. The correct usage is "maximize" in conjunction with "neg_mean_squared_error". We know this because the sklearn documentation says so, and also because we can demonstrate it with a simple script.
In every machine learning problem, the task is to minimize the error. One way to measure error in regression is to take the square of the difference between the observed data and the model's predictions, i.e. minimizing $(y - \hat{y})^2$.
On the other hand, the package sklearn adopted the convention of maximizing a "score". In skelarn's parlance, a "score" is an objective function that you want to maximize. From the documentation:

For the most common use cases, you can designate a scorer object with the scoring parameter; the table below shows all possible values. All scorer objects follow the convention that higher return values are better than lower return values. Thus metrics which measure the distance between the model and the data, like metrics.mean_squared_error, are available as neg_mean_squared_error which return the negated value of the metric.

We can verify that sklearn works as described in the documentation with a simple script.
from sklearn.datasets import load_boston
from sklearn.linear_model import Lasso
from sklearn.model_selection import cross_validate

X, y = load_boston(return_X_y=True)

lasso = Lasso(random_state=0, max_iter=10000)

foo = cross_validate(lasso, X=X,y=y,scoring="neg_mean_squared_error")
print((foo["test_score"] < 0.0).all())

Which prints True. This shows that the return from cross_validate with "neg_mean_squared_error" must be the negative of the square error (because squaring a real number is non-negative). If you negate the function that you're minimizing, the minimization  transforms into a maximization. Therefore, the code in the question is maximizing model misfit.
A: This was an interesting example. Here is some R code for some naive analysis on Boston Housing data (predicting the median value MEDV):
library(mlbench)
library(glmnet)
data("BostonHousing")

n = nrow(BostonHousing)
train_idx = sample(n, round(0.7*n))
train_data = BostonHousing[train_idx, ]
test_data = BostonHousing[-train_idx, ]
lmod = lm(medv ~ ., data = train_data)
mean(lmod$residuals^2) # training MSE for the linear model

r_col = ncol(train_data) # which column contains the response -- here it is the last column

lasso_mod = cv.glmnet(x = data.matrix(train_data[, -r_col]), y = train_data$medv, nfolds = 5)
lasso_mod
ridge_mod = cv.glmnet(x = data.matrix(train_data[, -r_col]), y = train_data$medv, nfolds = 5, alpha = 0)
ridge_mod

loss = function(x,xh) mean((x-xh)^2)

loss(predict(lmod, test_data[,-r_col]), test_data$medv)
loss(predict(lasso_mod, data.matrix(test_data[,-r_col])), test_data$medv)
loss(predict(ridge_mod, data.matrix(test_data[,-r_col])), test_data$medv)


plot(lasso_mod)
plot(ridge_mod)

The numbers for the test MSE of the linear model, the Lasso and the Ridge are 26.7, 33 and 32.2. The latter two are the default models selected by the glment. These numbers are quite variable since this is based on a single training/test split, however, the order seems to be roughly the same. At least the linear model always does better (in my limited set of runs).
Here are the regularization curves produced by the glmnet package:
For the Lasso:

For the Ridge:

They do suggest that the best regularization is essentially no regularization ($\lambda = 0$) which is consistent with the results.
Do you always have to get a better result with Lasso and/or ridge? No if $\lambda = 0$ is the best choice.
What can you do? Perhaps try to be more refined in what variables to include in the model and better take care of the categorical variables. The above analysis does it naively treating categorical variables as integers which is not that meaningful. In the end, there is no guarantee that Lasso or ridge do better than the linear model.
You can try some non-parametric regression models too. You can also do polynomial regression coupled with lasso to reduce the complexity.
PS. The training MSEs for the above setup are 20.8 for the linear model and the following for Lasso:
Measure: Mean-Squared Error 

    Lambda Index Measure    SE Nonzero
min 0.0174    65   24.78 4.346      13
1se 0.9505    22   28.85 5.214       6

and for the following for ridge:
Measure: Mean-Squared Error 

    Lambda Index Measure    SE Nonzero
min  0.671   100   22.91 2.166      13
1se  3.578    82   25.03 2.229      13

