Random forests regression better than OLS out of sample I am running an OLS regression on panel data. The dependent variable is the governement yield of several countries. The independent variables are several macroeconmic variables (i.e. inflation, unemployment, etc.). My data consists of 10 years of data for several countries (100 observations in total).
I am using the LSDV method for Fixed effects. In the end, I obtain an adjusted R-squared of around 85%.
On the other hand, I ran a random Forest using the same data. Here, I obtained an R-squared of 60%. I know these two R-squares are not really comparable. Therefore, I did a cross validation. I prepared a training set (first 8 years) and a test set (last 2 years). Then, I ran both models (OLS and random forests). The Random Forest regression outperforms OLS in the test set (using RMSE). However,RF seems to underperform insample.
Have you had any similar results in the past?
Thanks
Jonas
 A: This is not surprising, but you should be careful running a random forest on de-meaned (within transformation) data.
Start with a basic model 
\begin{equation}\label{basic}
y_{it} = \alpha_i+\mathbf{X}_{it}\beta + f(\mathbf{Z}_{it}) + \epsilon_{it}
\end{equation}
where $y$ is an outcome for unit $i$ in time $t$, $\mathbf{X}$ is a $N\times P_X$ matrix of data to be represented linearly, and $\mathbf{Z}$ is a $N\times P_Z$ matrix of variables to be represented nonparametrically.  All or some elements of $\mathbf{Z}$ may be included in $\mathbf{X}$, yielding a linear ``main effect'' along with a nonlinear component.  The compound error $\alpha_i + \epsilon_{it}$ represents between-unit and within-unit variability, respectively.
Rather than estimating hundreds or thousands of individual effect parameters, standard econometric practice removes $\alpha$ via the ``within'' transformation:
\begin{equation}
y_{it}-\bar{y}_i = \left(\mathbf{X}_{it}-\bar{\mathbf{X}}_{i}\right)\beta + f^*(\mathbf{Z}_{it}-\bar{\mathbf{Z}}_{i}) + \epsilon_{it}- \bar\epsilon_{i}
\end{equation}
This fails however when $f()$ is nonlinear, because $f(\mathbf{Z}_{it}) \neq f^*(\mathbf{Z}_{it}-\bar{\mathbf{Z}}_{i})$.  While multidimensional linear basis expansions of $f(\mathbf{Z})$ can solve this problem, they are quickly overcome by the curse of dimensionality as $P_Z$ grows.  
Many machine learning approaches are ill-suited to estimating this model.  Random forests can consistently and efficiently estimate models of the form $y = f(\mathbf{Z}) + \epsilon$, but they have no clear way to incorporate known parametric structure where available, nor an obvious way to eliminate the nuisance parameters $\alpha$.  Elastic-net regression and generalized additive models} can handle fixed effects by first projecting-out fixed effects, but they require extremely large multidimensional basis expansions if they are to serve as universal approximators of an arbitrary $f()$ with a high-dimensional $\mathbf{Z}$.  
If you can, you want to find an approach that can estimate the fixed effects jointly with the nonparametric component.  I've attempted to do so using neural networks, but this work is still very much in progress.  
