6
$\begingroup$

I am training a XGBoost regression model for predicting number of applications and the range of the target variable in train and test data set is different. For e.g:

  • In Train data : Minimum applications = 40 Maximum applications = 1500
  • In test data : Minimum applications = 400 Maximum applications = 600

Obviously the standard deviation is also different for the test and train datset

Hence the model is getting confused while predicting the value. Any suggestions on how should I solve this issue?

$\endgroup$
4
  • 2
    $\begingroup$ Is there a reason that the test and train have different distributions? They should really be samples from the same overall population for the testing to make sense. $\endgroup$ – jcken Mar 22 at 16:53
  • $\begingroup$ So due to covid the business took a hit and now since things are back in action, the number of applications seems to have jumped and thus we are seeing the minimum applications to start from 400 $\endgroup$ – Anubhav Nehru Mar 22 at 17:01
  • 2
    $\begingroup$ Some related discussion about accounting for the effects of covid in modeling efforts: stats.stackexchange.com/questions/514358/… $\endgroup$ – Sycorax Mar 22 at 17:06
  • $\begingroup$ Can someone explain why this is inherently an issue? If the goal was to predict daily high temperature, it's easy to imagine a dataset wher a range from 0-100F is observed in a multi-year train set while only temperatures in the range of 40-60F are observed in a test set which corresponds to a single month. If we created a model that acurately predicted temperature, can't we still apply it? Are we assuming that because the range of the response changed, the relationship between predictors and response also changed? $\endgroup$ – Ryan Volpi Mar 22 at 18:42
2
$\begingroup$

Collect data from the population you want to predict, and use that data to train your model. This is the best approach.

Alternatively, think long and deep about how you can possibly post-process your predictions (from a model trained on a sample from population A) to apply to population B. How you do this will depend on your subject matter. The simplest way, based on the information you give, would be to apply an affine linear transformation to your predictions, to map the interval $[40,1500]$ to $[400,600]$.

Given that you already have your model, a simple approach like this will likely be cheaper and faster than collecting new data from the new population, so it may make sense to try it and to think about whether you can realistically hope that a new model will be sufficiently better to justify the added expense of new data collection.

$\endgroup$
6
  • $\begingroup$ Hi @Stephan, thanks for your answer. In my problem , the training data starts from the month of June 2020 till Jan 2021 and I am testing the model on the data for the month of Feb 2021, hence I am unsure how to go with your solution. Kindly give more inputs from your end $\endgroup$ – Anubhav Nehru Mar 22 at 17:10
  • 1
    $\begingroup$ Accounting for COVID (or any ongoing structural break is hard). You will need to use your domain knowledge, and make assumptions on whether the new level of applications is a "new normal", or whether the level will come down again. I understand that you can't train a model on the new data, because it doesn't exist yet. Right now, it may be that the best bet is really to take your predictions and scale them affine-linearly to a more reasonable range. Alternatively, consider setting a boolean predictor that captures the COVID effects. $\endgroup$ – Stephan Kolassa Mar 23 at 6:49
  • $\begingroup$ In theory expert judgement, or if you have lots of time Delphi method, might deal with a structural break (although I doubt anyone would ever do the latter). One thing that concerns me in the above example is seasonality because that is very common in the data I work with. If this changes from the year you train with than I would think you have a major problem. $\endgroup$ – user54285 Mar 25 at 2:02
  • $\begingroup$ @user54285: you are certainly right. One can in principle model structural breaks in any predictor's influence, although "structural break" in everyday usage only refers to shifts in the mean, i.e., in the influence of the intercept. But, e.g. in a regression framework, we can include interaction terms between the structural break regressor and other regressors, which would model changes in trend or seasonal (or any other) behavior. The problem now is how to estimate that with very limited data... Bayesian approaches may be helpful. $\endgroup$ – Stephan Kolassa Mar 25 at 6:55
  • $\begingroup$ @stephan Kolassa thanks for your answer. Is the structural break regressor something that would be say 0 before an intervention and 1 on or after (or a structural break I guess rather than an intervention). $\endgroup$ – user54285 Mar 25 at 21:01
2
$\begingroup$

Like I said in my comment, I don't necessarily see how a change in the range of the response is an issue. In particular, in your case, I think it is a red herring.

First, it's important to ask, should the test set have the same range of responses as the train set? In many cases it is not the case and I don't know why that would be problematic necessarily. If the goal was to predict daily high temperature, it's easy to imagine a dataset where a range of temperatures from 0-100F is observed in a multi-year train set while only temperatures in the range of 40-60F are observed in a test set which corresponds to a single month. If we created a model that acurately predicted temperature, we should still expect it to perform well on this test set assuming we have a good model in the first place.

Some exceptions include

  1. If the relationship between the predictors and the response changes (concept drift).

  2. If the domain of the data shifts away from the domain of the training data (applicability domain). In this case, your model has to extrapolate to a region it has never seen before. (See Applied Predictive Modeling pg 535) In this example, we don't know anything about the range of the predictors, but the range of the response for the test set is clearly within the range of the response we see in the train set.

If the goal is to estimate how well the model will perform when applied, it makes sense to split the train and test sets based on time and evaluate performance on future months. If there are minor performance losses related to shifts in the distribution of the response, then that is normal and part of what you want to estimate. That being said, it would be better to evaluate the model over a longer period, or over multiple one month periods using time series cross-validation (See Forecasting: Principles and Practice, Section 3.4). That is, unless you're only concerned with how it will perform in Februay, sepcifically.

If I had to guess, the reason you are not getting acceptable performance has nothing to do with the range of the response. One, more likely, cuplrit is omitted variable bias. In particular, the effects of covid are very likely at play and should somehow be accounted for (see the comment by @Sycorax). Additionally, without knowing anything else, I might imagine the quanitity of 'applications' to have some seasonality. Both of these factors will complicate the analysis, especially when you do not have much data.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.