Training twice on non-injective data I have a large dataset of 30000 points, but most my Ys are the same while all Xs are different. Ys are from different samples, so I had means of Y for each sample and I used means alongside Xs to train an RF. Then I used the predicted values by this RF as a new Y to train another RF that uses only Xs as input. This surprisingly resulted in a 20 percent decrease in RMSE (final predicted Y vs initial non-injective Y).
So to cap it:
X + mean of Y for samples vs Y using RF
then
X vs YpredictedByFirstRF using RF
Did I commit some kind of machine learning crime? My idea was to simulate noise by adding central tendency measures of output as input for first model.
 A: IIUC, instead of training a random forest directly on your given data pairs $(x_i, y_i)$, you first transformed each of those pairs into another pair $(x_i, y^\prime_i)$ (using some sub-means of the $y_i$ and a random forest model $R_1$) and then you trained a second random forest model $R_2$ on this transformed dataset $\{(x_i, y^\prime_i)\}_i$. So one could say that you used "augmented data" for training the second random forest.
Training models on augmented data instead of the (inferior) original data is common practice and there is lots of literature on how to augment datasets.
However, it would of course be important to compute the RMSE of $R_2$ on a test dataset $\{(x^t_i, y^t_i)\}_i$, one that has not been used in your augmentation procedure and in training your random forest $R_2$. It is not clear from your question how and on what data you measured this 20% decrease of RMSE.
As to why your approach works: I am a surprised that this works and I don't see a reason why this should work in general, especially with such a huge improvement of 20%. Maybe it is a speciality of your data. I would suggest some variations to figure out what is going on:

*

*Does this method also work with other data?

*Do the same without the extra y-means, to make sure that it is really the $y$ means that you provide that make the difference.

*Substitute other models for random forest, e.g. GBMs, neural networks, ... Can those models also be improved by your method?

*Iterate the method several times. Does it keep improving?

