Applying machine learning techniques to panel data I have a panel data in which I observe 1500 companies and many individuals work  for those companies for multiple periods. I have explanatory variables at both individual (e.g. race, age, education) and company level (e.g. company age, R&D investment, spending on advertising, industry). So there are different types of explanatory variables i.e. continuous, categorical, binary. In this dataset, same individual might work for more than one company at the same time (given that some of them are consultants). My dependent variable is sales per year. 
By using this data, I want to make a prediction of the dependent variable  and want to test the importance of each explanatory variable. Does anyone know which models would be more suitable and where I could find a reliable material on this topic? I was thinking about applying RNN to panel data (how to do it ?) but also open to other suggestions.
I know still ML and econometrics are not talking to each other with regard to causality but do you know any recent paper/ development related to this issue?
 A: I don't believe I can offer what you're looking for, but the first step is to use the repeated individual_id as a variable to ensure that each individual is in 1 partition. For example if you're using cross-fold validation, then an individual should only show up in 1 fold and not be spread out amongst the others. 
As far as what machine learning algorithms to try - that is ultimately up to the data. In my experience though, I think your best results will come from some sort of boosted tree such as LightGBM or xGBoost. This will lead to you deciding how to encode the categorical variables, for which I recommend category_encoders library in python, if you're using python.
I'm sure there's interesting and novel ideas around RNN's but to be honest I don't think this problem is suited for that type of algorithm. This sounds like a classic regression problem to me.
A: Adding to Josh that using some tree based algorithm is lacking some clear transparency on how the features are impacting your sales (like you have in a classic linear regression framework, which is called a white-box model, because the coefficients are very easy to intertrep).
What you can get as relative importance from these tree based algorithms, which basically tell you how important the individual features are for constructing your tree and the output is a ranking. But with this ranking you can not get an understanding of how much an increase of x impacts your y.
There are two algorithms out there, which I know exist but I have never use so far:

*

*Causal Forest is like a tree based algorithm, but incorporates the idea of causal inference and can be found here: https://lost-stats.github.io/Machine_Learning/causal_forest.html


*Google's What-If Tool (WIT) is designed to follow the idea of counterfactuals, which can answer the question what would have happened if...


*You can build your own counterfactuals using predictions. Train a model and get your predictions for varying values of the individual features (c.p.) and see how the output changes. Like this you can have an understanding which features impact the y how much even using some black-box models like neural nets or boosted trees.
Lastly I also want to mention that using scikit-learn you can return PDP (partial dependency plot) and ICE (individual conditional expectation) plots, which can give you an understanding of the relationship between your feature(s) and your target.
