Regression with small target variable interval

Question

I am trying to train a Gradient Boosting on a '%-target variable', i.e. having values in the interval [0,1]. The bad thing about this particular case is that the target variable is very narrowly distributed around the value 0.99. It is not constant, there are different values, it is just that they all lie very close to 0.99. Running a usual Gradient Boosting gave me a constant model, i.e. all the trees were degenerate to one single node predicting just one number.

Question: How to force Gradient Boosting (or, more generally, any regression model) to become non-degenerate when the target variable values lie very very close to each other?

Conceptually the model is right (it has a small 'absolute' error) so this is what I tried/thought about so far:

1. Apply preprocessing step that 'pulls apart' different values of the target function
2. Just multiply the cost function by a constant

On 1.:

Surprisingly, 1. did not really work. I tried to linearly scale the target variable so that the min of all of the values (0.989) becomes 0 and the maximal value (0.999) becomes 1. I also tried a 'smoother' version of that where I rescaled all the boundaries of different quantiles, etc. However I can not see any obvious mistake in the code but the model does not perform well. Even worse: When I test this on an other target variable where the already exists a working model, the performance totally drops when I apply this preprocessing step (and of course, I rescale the prediction in the end :-)) while without this step, the model performance is fine.

Question (2): Aren't trees supposed to work with splits and so on? It should not make much of a difference in a regression task whether I use the original target variable or a scaled version of it, right?

Maybe it is due to the fact that we use such a weird loss function (cross entropy) for logistic regression?

On 2.:

Question (3): Does anybody know how to easily scale the cost function in xgboost?

Is that even a valid approach? Doesn't this only mean that the gradients will grow with that constant?

Answer 1

In case other people may be interested: I think I figured out how to deal with such a situation.

The way it worked for me: Just use a translation (i.e. add a constant value $c=(0.5-\text{mean of target variable})$ to the target column) instead of a rescaling. That step effectively moves the mean of the data to 0.5. The reason why that works is that the squashing function (the sigmoid function) has the highest slope at that point, i.e. predictions with very little differences will cause big changes in the loss function. That is actually what I wanted to do with the scaling: I wanted the model to be focus more on 'small differences'.

Why does scaling hurt so much? Still no answer... I am unable to explain...

Even if scaling did not hurt so much, why didn't it work? No explanation still... I guess that the scaling interferes too much with the sigmoidal function somewhat...

Hope this helps and please leave a comment or an answer I can accept if you can answer the other questions...