This is not surprising, but you should be careful running a random forest on de-meaned (within transformation) data.
Start with a basic model
\begin{equation}\label{basic}
y_{it} = \alpha_i+\mathbf{X}_{it}\beta + f(\mathbf{Z}_{it}) + \epsilon_{it}
\end{equation}
where $y$ is an outcome for unit $i$ in time $t$, $\mathbf{X}$ is a $N\times P_X$ matrix of data to be represented linearly, and $\mathbf{Z}$ is a $N\times P_Z$ matrix of variables to be represented nonparametrically. All or some elements of $\mathbf{Z}$ may be included in $\mathbf{X}$, yielding a linear ``main effect'' along with a nonlinear component. The compound error $\alpha_i + \epsilon_{it}$ represents between-unit and within-unit variability, respectively.
Rather than estimating hundreds or thousands of individual effect parameters, standard econometric practice removes $\alpha$ via the ``within'' transformation:
\begin{equation}
y_{it}-\bar{y}_i = \left(\mathbf{X}_{it}-\bar{\mathbf{X}}_{i}\right)\beta + f^*(\mathbf{Z}_{it}-\bar{\mathbf{Z}}_{i}) + \epsilon_{it}- \bar\epsilon_{i}
\end{equation}
This fails however when $f()$ is nonlinear, because $f(\mathbf{Z}_{it}) \neq f^*(\mathbf{Z}_{it}-\bar{\mathbf{Z}}_{i})$. While multidimensional linear basis expansions of $f(\mathbf{Z})$ can solve this problem, they are quickly overcome by the curse of dimensionality as $P_Z$ grows.
Many machine learning approaches are ill-suited to estimating this model. Random forests can consistently and efficiently estimate models of the form $y = f(\mathbf{Z}) + \epsilon$, but they have no clear way to incorporate known parametric structure where available, nor an obvious way to eliminate the nuisance parameters $\alpha$. Elastic-net regression and generalized additive models} can handle fixed effects by first projecting-out fixed effects, but they require extremely large multidimensional basis expansions if they are to serve as universal approximators of an arbitrary $f()$ with a high-dimensional $\mathbf{Z}$.
If you can, you want to find an approach that can estimate the fixed effects jointly with the nonparametric component. I've attempted to do so using neural networks, but this work is still very much in progress.
