I'm doing a project about the effect of synthetic oversampling in a machine learning context (more precise SMOTE for the oversampling of the minority class of a highly imbalanced target variable). The dataset is about historic banking defaults from the last several years. The imbalance problem obviously stems from the fact that banking defaults are comparably rare. Concerning this, I've two related questions, the main one dealing with the appropriate point in the data mining pipeline for the oversampling process, the other one about the right data format for such panel data in a machine learning context.

Short description of the dataset: for each quarter over a time span of several years I've measurements about each bank, for example, the return on assets. Furthermore, I'm provided with the status of each bank in every period, which can either be 0 or 1 for a bank going into default or operating successfully. For a single point in time $t$, the goal is to predict the status of a bank 2,4,6 or 8 ($h$ - prediction horizon) quarters ahead. Originally the data comes in a long format, each single observation representing one bank in a certain period described by around 15 different measurements. The dependent variable is the performance (0/1) at one point $t+h$, the features or independent variables are every available measurement until $t-1$. The data is delivered in a long format in the following way:

\begin{array}{|l|l|l|l|l|} \hline \text{bank id} & \text{period} & \text{roa} & \dots & \text{target} &\text{overallFailure} \\ \hline 1 & 1 & & & 0 &0 \\ \hline 2 & 1 & & & 1 &1 \\ \hline \vdots & \vdots & & & 0 & \\ \hline 100 & 1 & & & 0&0 \\ \hline 1 & 2 & & & 0 &0\\ \hline 2 & NA & NA & NA & NA&1 \\ \hline \vdots & \vdots & & & 0& \\ \hline 100 & 2 & & & 0&0 \\ \hline \vdots & \vdots & \vdots &\vdots & \vdots&\vdots \\ \hline \end{array}

So each row or observation comes as the measurement of one bank at one point in time. After a bank goes into default it's filled with NA from the following period on. The last variable, let's call it $\verb+overallFailure+$, captures if a bank is going into default at ANY point in time and takes the same value in every period.

1) The first question might be stupid but is rather about the appropriate data format. My approach would actually be to bring the data frame to a wide format. Every single data row would represent one bank, each of the columns representing a measurement at one point in time. Same holds for the target variable at every point in time. I would say I'm more or less familiar with machine learning approaches in general but I'm not sure how to implement the lagged information here.

2) The crucial point, related to question 1 is actually the right time for applying the SMOTE algorithm during the pipeline. Let's say all the features I'm going to use are more or less delivered and should all be used in the model, so I don't have to worry about feature selection or feature engineering. The experiment is more about the effect of different oversampling rates than about a perfect model. It's clear that I have to apply SMOTE after splitting into training/test and ONLY to the observations the model is trained on. Before applying it, ALL observations should be normalized by subtracting the mean of the training sample and dividing by its variance. Hyperparameters (like the $k$ and the oversampling rate) should be optimized via cross validation. I'm wondering about the fact that I've several different classification problems I'm looking at. The problem in point $t$ looking at $t+h$ is obviously different than standing in point $t+1$ looking at $t+h+1$ and so might be the distribution of the target variable. There are periods with no minority observation at all, meaning no input for the SMOTE algorithm. At which point should the oversampling actually take place?

  • $\begingroup$ This doesn't answer your question (and admittedly, I'm a hard skeptic of methodologies like SMOTE), but I want to mention that something like bank default prediction lends itself much more to a risk estimation context than a classification context. SMOTE is intended to fix up methodologies that do not work by estimating probabilities of default, i.e. do not estimate risk, but this is what you want to do here! Of course, the probability of default output from your model will always be small, since the prior probability of default is small, but that's just the true nature of the problem. $\endgroup$ – Matthew Drury Aug 23 at 16:49

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.