Identify suitable scoring metric for food prediction I am using GridSearchCV to find the best parameter that help me tune XGBoost for a food prediction algorithm.
I am struggling to identify the best scoring metric that would result in the best profit (sales margin minus wastage costs) as this is ultimately what I am looking for. In running the script below and plugging it into the data (I reserved some data for testing only), I noticed that a higher R Square seems to be  better than a lower RMSE in obtaining a higher profit. But I am struggling to find an explanation which will help me guide to the best scoring method.
Here some infos on the situation: It costs me 6 USD to produce the product and 9 USD to sell, so my margin is 3 USD. Therefore my wastage is 6 USD multiplied by (production minus sales quantities), whereas my earnings are sales quantities multiplied by 3. Example: I produce 100, sell 70, waste 30 my earnings are 70*3 - 30*6 = 30 So I have an imbalance between sales and wastage.
Main Question: Which scoring metric puts a higher penalty weight on the over-prediction?
My current code:
My current code:
X = consumption[feature_names]
y = consumption['Meal1']

data_dmatrix = xgb.DMatrix(data=X,label=y)

# Create the parameter grid: gbm_param_grid
gbm_param_grid = {
    'min_child_weight':[1, 2],
    'gamma': [0.05,0.06],
    'reg_alpha':range(1, 2),
    'colsample_bytree': [0.22, 0.23],
    'n_estimators': range(28, 29),
    'max_depth': range(3, 8),
    'reg_alpha':range(1, 2),
    'reg_lambda':range(1, 2),
    'subsample': [0.7,0.8,0.9],
    'learning_rate': [0.1,0.2],


}
fixed_params = {'objective':'reg:squarederror','booster':'gbtree' }

# Instantiate the regressor: gbm
gbm = xgb.XGBRegressor(**fixed_params)

# Perform grid search: grid_mse
grid_mse = GridSearchCV(estimator=gbm, param_grid=gbm_param_grid, scoring="r2", cv=5, verbose=1)


# Fit grid_mse to the data
grid_mse.fit(X,y)

# Print the best parameters and lowest RMSE
print("Best parameters found: ", grid_mse.best_params_)
print("Lowest Score found: ", np.sqrt(np.abs(grid_mse.best_score_)))

 A: You apparently have a situation that fits the newsvendor model in the inventory control literature: you produce for each period, face an uncertain demand, sell as much as is demanded or as much product as you have (whatever is lower), and throw away any leftovers.
Let $p$ denote your selling price and $c$ your purchasing or production cost, and assume your demand follows some distribution with CDF $F$. Then the newsvendor model tells you that your optimal stock position, in terms of expected profit, is $F^{-1}\big(\frac{p-c}{p}\big)$. In the situation you sketched above, $p=9$ and $c=6$, so you are interested in
$$F^{-1}\bigg(\frac{9-6}{9}\bigg)=F^{-1}\bigg(\frac{1}{3}\bigg).$$
In prediction or forecasting, this quantity is a quantile forecast and is more commonly denoted by 
$$ \hat{q}_{\frac{1}{3}} := F^{-1}\bigg(\frac{1}{3}\bigg).$$
One problem is that your distribution $F$ is almost certainly not the same in the entire forecasting horizon, $F_t$, because of time dynamics like seasonality or promotional efforts, so we need to cope with time effects,
$$ \hat{q}_{\frac{1}{3}}(t) := F_t^{-1}\bigg(\frac{1}{3}\bigg).$$
Now, how do we elicit this particular quantile forecast? It turns out (Gneiting, 2011, "Quantiles as optimal point forecasts", International Journal of Forecasting) that there is a simple loss function which is minimized by the correct quantile forecast, namely
$$ \text{PL}_\alpha(\hat{y},y) := \begin{cases}
\alpha|\hat{y}-y|, & \text{ if }\hat{y}\leq y \\
(1-\alpha)|\hat{y}-y|, & \text{ if }\hat{y}> y.
\end{cases} $$
In your case, $\alpha=\frac{1}{3}$. Note that this is precisely the loss function used in quantile-regression.
One problem is that this function is not everywhere differentiable. However, it can be approximated smoothly to any desired accuracy (Zheng, 2011, "Gradient descent algorithms for quantile regression with smooth approximation", International Journal of Machine Learning and Cybernetics).

Now, a number of caveats apply.


*

*Does a newsvendor model truly describe your data well? Are your products truly as highly perishable as (paper) newspapers? Or can product be sold over multiple periods? If so, you might need to do some more analysis, probably including simulation of quantile forecasts for different quantiles (note the impact of FIFO/LIFO behavior of your customers).

*Does a service level of only 33% really make sense? Depending on your demand distribution, that may mean a lot of empty shelves at the end of the day, and of frustrated customers. (And also of possibly censored demands once your system goes live. If you decide to live with that, you will need to account for the censoring when you retrain your model.) You may need to think about whether the costs of stockouts are adequately captured by the lost sales alone.

