0
$\begingroup$

I have a prediction task, in which I use DecisionTreeRegressor of scikit-learn to predict a target label, which is about a certain user behaviour in a web platform (and it has a range of 0-4). The features are generated based on users' other activities in the web platform.

I have separate training and test sets. The training set is from the 2nd week activities, and the test is from the 4th week activities of the users. So, I want to train a model using the 2nd week data, and test it on the 3rd week. In both sets, the target labels are imbalanced. The reason for the imbalance is that users are encouraged to participate at a certain level, which is 3 times. Thus, in both sets there is an accumalation at 3. For example, the number of samples with 3-times participation is 400 whereas the number of users with 1-participation is 65, and the number of users with 0-participation is 55.

To obtain a balanced target labels in the training set, we oversampled it to have equal numbers at each participation level (e.g., 0-participation:250, 1-participation: 250, 2-participation:250, 3-participation:250, 4-participation: 250). Just to explore, splitting the training set into train & test, the prediction results are very good (Mean absolute error is around: 0.20) -See Figure 1.

After we trained the model (using the whole training set), we make predictions on the test set (which is imbalanced itself), the results do not seem to be as promising (Mean absolute error is around: 0.55) -See Figure 2. When I oversample the test set as well, the prediction performance worsens (MAE increases to 0.80) -See Figure 3.

The figures actually tells the story:

Figure 1

enter image description here

Figure 2

enter image description here

Figure 3

enter image description here

At this point I do not know how to proceed. So, I should just go with the results in Figure 2, and discuss the effects of external factors (being required to do 3-times) on user behavior. This is because no matter users have different activity patterns (which were used to generate features), they may just participate on an activity because they are required. I wonder what would be a good approach to understand these results. This is going to be for an academic work.

$\endgroup$
2
$\begingroup$

There is a lot going on in here and it's hard to say what your actual question is, but here are two pieces of general advice:

First, you probably don't have enough data that it's safe do do train-test-validate splitting. Your results will probably vary depending on which samples end up in the test set. Unless you have many thousands of data points and a reasonable signal-to-noise ratio, you should look into cross-validation or even resampling methods such as bootstrapping.

Second, models fit on artificially balanced data won't generalize to the unbalance population. "Balanced data" isn't all it's cracked up to be. Unless you have a really good reason to oversample and a principled way to correct for this oversampling, you should stay away from artificial balancing.

$\endgroup$
  • $\begingroup$ thanks for the advice, Well, in my case then you would prefer resampling rather than oversampling, is that correct? So that I can have more sample for each target label although although the data remains still imbalanced? Also, then, when are we supposed to use these artificial balancing techniques? $\endgroup$ – renakre Apr 24 '17 at 10:31
  • $\begingroup$ @renakre yes, I would probably do bootstrapping the get a stable error estimate and to get a sense of its uncertainty. To be honest I can't think of many situations where artificial balancing makes sense. One solid example is case--control studies where it is hard to get more cases, but you can get as many controls as you want. $\endgroup$ – einar Apr 24 '17 at 10:43
  • $\begingroup$ @renakre PS: another source of variance for you is probably the use of decision trees, which are often high-variance $\endgroup$ – einar Apr 24 '17 at 10:45
  • $\begingroup$ could you please elaborate on this more? You mean decision trees may not be a good fit for my study? $\endgroup$ – renakre Apr 24 '17 at 10:52
  • $\begingroup$ @renakre That really depends on too many things for me to be able to answer well. I'm just saying a small change in data can result in a very different-looking tree, which might be another reason you're seeing results that are hard to interpret $\endgroup$ – einar Apr 24 '17 at 10:57

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.