Poor model performance on certain out-of-sample data We're noticing poor model performance on certain out of sample products.

*

*We have trained a ML model on about 2000 different products in a few markets.

*Our predictors include
a) product characteristics
b) historical sales, revenue, marketing expenses etc.
c) Location and demographic control variables such as median income

Target variable is price / value of the product.
When we fit the model and calculate our overall error, it's about 15%. For certain properties, the ones that are very high in value / prestigious or in sparse locations, the model doesn't do a good job of prediction. The error is quite high 25-40%.
Additional info:


*Is this a time series problem?

We do not have time series data, but, we refresh the data monthly.


*How much data (rows) do you have?

2000 rows * 15 columns (numeric and categorical)


*Are you training a single model on all different products?

Yes, single model for different products with sales data from different markets. Location/market-specific variables are included in the model.


*Are you using cross validation to tune your model? How exactly are you doing your train/test split?
X_train, X_test, y_train, y_test =  train_test_split(df.iloc[:,0:14], df.iloc[:,-1], test_size=0.1,  random_state=42)
Training an xgboost model with RandomizedSearchCV.
model = RandomizedSearchCV(model, param_distributions=params, n_iter=50, scoring="neg_mean_absolute_error", cv=5, verbose=1, n_jobs=1, return_train_score=True, error_score='raise')

where params,
params = {"objective":"reg:squarederror", 
          "colsample_bytree": xgbest.colsample_bytree, # fraction of cols to sample
          "gamma": xgbest.gamma, # min loss reduction required for next split
          "learning_rate": xgbest.learning_rate,
          "max_depth": xgbest.max_depth, # controls model complexity and overfitting
          "n_estimators": xgbest.n_estimators,
          "subsample": xgbest.subsample} # % of rows to use in training sample

We need to handle this variance in our model and lower the error rates. What are some ways to reduce the variance error of our model? Is this a case of model overfitting and how might we tackle this problem?
 A: Thank you for clarifying the information in the comments.
Here are my thoughts on your problem.
I think you probably have 2 issues (which are related to one another), which give you this kind of result.
Different variable relationships + not enough examples
First of all, I am guessing that the variable relationships between your predictors and low value items is not the same as the relationships between predictors and the high value items that the model struggles with.
As an example, maybe median income is very important for predicting low-value items (because median income determines how much people spend per month on necessities vs. luxury items), but is a very poor predictor of price for high value/rare/prestigious items because these items are purchased by the top 1% (who isn't reflected by the median income) or people who are super interested/collectors (who spend regardless of median income). In this case, median income doesn't matter much. This is just an example, but my point is that the "true" predictors for the high value items might be quite different from the "true" predictors for the low value items. Maybe you don't have enough of these "good" predictors for the high value items.
You might ask: "But I'm confident I have variables that are good for predicting high-value items, why doesn't the model learn that the variable relationships might be different?"
Here's where the second issue comes in. Think about the fact that the model values all your data points equally. For the model, getting a good prediction on a low value item is just as valuable as getting a good prediction on a high value item. I suspect it might be that your dataset has a lot LESS of these high-value item examples. So what might be happening is that the model simply doesn't have enough data to learn that these high value items have different characteristics. The model acts in this way: "Why should I try to optimize my predictions for this weird 5% of the data (and sacrifice performance on other 95%), when I can get good results on the other 95% and ignore these 5%". So, your high value item relationship might just be drowned out/buried because there aren't enough examples for it to be worth it for the model to try to optimize them (because average error is lower if model gets really good at 95% of data and "ignores" 5%).
In general, having 15 different variables and using an algorithm as complex as XGBoost, I would be surprised if 2000 rows was enough (especially if there's a lot of complexity/diversity in the dataset).
Is the model overfitting the data?
You need to compare the error in cross validation to the test-set error. If there's a big difference, then your model is overfitting. You might want to even look specifically at the cross validation error ONLY on high value items and compare that to the test set error of only the high value items. If there's a big difference, then the model is overfitting on that subset of the data (likely because there's not enough examples, but that's my guess).
I hope this provides you with some perspective and ideas for how to think about this problem.
EDIT: kms told me in a comment that indeed, it's an unbalanced dataset. So, I'll be tacking on some potential ways to work with that.
Here are 2 broad options when it comes to working with imbalanced data:

*

*Oversampling/undersampling - creating a new dataset which doesn't have the imbalance problem, therefore forcing the model to contend with the rare category of data

*Weighting - not changing the data, but changing the scoring so that the model is penalized differently fir getting different categories wrong.

These are big topics, so I won't be going into huge detail here, but I'll outline some areas to explore.
In addition, before I go on, the hands-down best thing to do is gather more data. So before engaging in any of the below, take a step back to consider whether you've really done everything you reasonably can to gather data. More real data is always the superior approach when it comes to imbalanced datasets (or any low-data-quantity setting).
Category 1: Sampling
Right off the bat, I (like many) would advise against the undersampling approach, where you would keep all of the rare category examples, but randomly sample some fraction of the "common" category samples to include. You already have a limited amount data and you want to make use of all of it. Undersampling gives your model just gives your model less information to work with. So if you're doing some sampling approach, I would go with oversampling.
Undersampling extreme example: You have 100 "common" cases and 10 "rare" cases, so you sample randomly 10 of the common cases and now you have a new, balanced dataset with 10 common cases and 10 rare cases.
Simply speaking, in over-sampling, you want to create a synthetic dataset where you have, in some way, repeated the rare data so that the model encounters more often and therefore has to address it in order to find the optimal solution (i.e. optimal solution can't be found by ignoring it anymore).
Oversampling extreme example: You still have 100 "common" cases and 10 "rare" cases, so you randomly sample (with replacement) from the 10 "rare" cases until you have 100 of them (of course, 90 of the 100 are randomly-selected repeats of the original 10). But now you have 100 "common" and 100 "rare" cases.
There's a technique called SMOTE, which you can read about here: https://machinelearningmastery.com/smote-oversampling-for-imbalanced-classification/ . I personally have not used it, but it's very well-known and considered to be quite effective (it's a specific kind of oversampling technique). If simple oversampling doesn't solve the issue, consider SMOTE.
Category 2: Weighting
This is pretty straightforward too. Besides making it more common, you can try to force the model to pay attention to the rare cases by assigning a larger weight to them - that is, the model is penalized more than usual when it gets them wrong. Many algorithms (including XGBoost, I believe) allow some kind of incorporation of weights into it (as parameters). I actually haven't encountered that in a regression problem, only classification, so to be honest, I'm not exactly sure how that would look. But you should certainly look into "sample weights xgboost", and see if you can find what you're looking for. Here's something to start you off: https://github.com/dmlc/xgboost/issues/520  (it at least has some key phrases that can help you find what you need).
