How to interpret results of a predictive model when an external factors leads to imbalanced target labels

Question

I have a prediction task, in which I use DecisionTreeRegressor of scikit-learn to predict a target label, which is about a certain user behaviour in a web platform (and it has a range of 0-4). The features are generated based on users' other activities in the web platform.

I have separate training and test sets. The training set is from the 2nd week activities, and the test is from the 4th week activities of the users. So, I want to train a model using the 2nd week data, and test it on the 3rd week. In both sets, the target labels are imbalanced. The reason for the imbalance is that users are encouraged to participate at a certain level, which is 3 times. Thus, in both sets there is an accumalation at 3. For example, the number of samples with 3-times participation is 400 whereas the number of users with 1-participation is 65, and the number of users with 0-participation is 55.

To obtain a balanced target labels in the training set, we oversampled it to have equal numbers at each participation level (e.g., 0-participation:250, 1-participation: 250, 2-participation:250, 3-participation:250, 4-participation: 250). Just to explore, splitting the training set into train & test, the prediction results are very good (Mean absolute error is around: 0.20) -See Figure 1.

After we trained the model (using the whole training set), we make predictions on the test set (which is imbalanced itself), the results do not seem to be as promising (Mean absolute error is around: 0.55) -See Figure 2. When I oversample the test set as well, the prediction performance worsens (MAE increases to 0.80) -See Figure 3.

The figures actually tells the story:

Figure 1

Figure 2

Figure 3

At this point I do not know how to proceed. So, I should just go with the results in Figure 2, and discuss the effects of external factors (being required to do 3-times) on user behavior. This is because no matter users have different activity patterns (which were used to generate features), they may just participate on an activity because they are required. I wonder what would be a good approach to understand these results. This is going to be for an academic work.

Answer 1

There is a lot going on in here and it's hard to say what your actual question is, but here are two pieces of general advice:

First, you probably don't have enough data that it's safe do do train-test-validate splitting. Your results will probably vary depending on which samples end up in the test set. Unless you have many thousands of data points and a reasonable signal-to-noise ratio, you should look into cross-validation or even resampling methods such as bootstrapping.

Second, models fit on artificially balanced data won't generalize to the unbalance population. "Balanced data" isn't all it's cracked up to be. Unless you have a really good reason to oversample and a principled way to correct for this oversampling, you should stay away from artificial balancing.