XGBoost classification of panel/longitudinal data observations I have a dataset of several firm quarters for a 10 year period (around 400 firms and around 30 observations per firm). For each quarter, there are a number of annualized financial ratios, which are used as features (around 40). Each quarter is associated with a binary label regarding a specific event happening or not in the following fiscal year. These events rarely occur, so my dataset is highly imbalanced. My main goal is to predict the probability of this event happening in the following year based on these ratios.
As such, I would like to be able to train based on financial quarters from the past (e.g 2008 to 2015) and classify quarter observations from 2017 (in order to find if this event occurs in 2018). 
As of now, I'm trying to train an XGBoost classifier, just using the financial data from observations (no time or firm variables), but my results are somewhat disappointing.
The time effects are irrelevant in my problem, but I'm afraid firm-specific effects may corrupt the data classification of observations since data samples of neighbor observations or firm observations are correlated.
Is there a better way to adapt an algorithm like XGBoost, that works well on imbalanced data, to a problem like this?
I tried some suggested approaches, such as training a classifier for each firm, but this approach does not translate over well to my problem since in a lot of firms the event I'm trying to predict does not occur. I also tried including dummies for each firm in my XGBoost model, but it doesn't seem to improve my detection results, and the one-hot encoding method dramatically the number of features, as I have more than 400 firms.
Thanks in advance.
 A: I've had a similar problem in my healthcare longitudinal dataset. My goal is to predict the probability of dropping out happening in the following month based on features during the current month.
My approach is adding features to represent longitudinality, e.g., current month since the 1st enrolled date (1-12) or time-varying age (float, not integer here) for each patient (since each patient will have <=12 rows of data. Why 12? I'm following a patient for a year, so 12 means 1-12 month, this is like a period). So now, each row represents each patient at a particular time point (e.g., during a month period), also a binary label (0/1) is associated with each row, an example would be like the following:




patient
start_month
# of response
other features
current_month
label




Mary
6
3

0
0


Mary
6
2

1
0


Mary
6
8

2
1


John
2
10

0
0


John
2
1

1
0


John
2
6

2
0


John
2
4

3
0


John
2
10

4
1


Cate
11
11

0
0


Cate
11
19

1
0


Cate
11
9

2
0


Cate
11
14

3
1


...
...
...

...
...




(As you can see, this would be a huge dataset since each patient will have multiple rows representing different time point)
You can see current_month depicts the longitudinality, and start_month remains the same per patient. One thing I need to clarify is that my label here is 1 ONLY at the last row per patient, since the patient can only "die" once, but you can set up label values based on your problem. The label is capturing the next period behaviour, e.g., the first 3 rows mean Mary did NOT drop out in her 2nd & 3rd month, BUT dropped out in her 4th month
After all above data wrangling, then fit this data to any classification algorithms and make binary predictions for each patient at a particular time point. It may encounter a high imbalance problem like my following dataset, just use any upsampling/downsampling method to solve it like SMOTE etc...
Not sure if this helps you, but I'm happy to discuss more. :)
