How to generate Feature vector from multiple tables of a database for classification

Question

Ive got 6 tables and each of them have multiple rows. The tables are :-

candidate_general_info, previous_employment_history, previous_health_issues, present_health_issues, family_health_history, present_claim_information

based on attributes whether to approve the claim or reject it.

1. candidate_general_info : This is a single row (age,sex,country etc)
2. previous_employment_history : This can be zero or more rows (person can be unemployed, child etc)
3. previous_health_issues : This can be zero or more rows
4. present_health_issues : This will be one or more (Someone had a health issue & thats what he is claiming for)
5. family_health_history : This will be zero or more
6. present_claim_information : This will be one or more (Someone had a health issue & thats what he is claiming for). Information about disease, claim amount, number of days hospitalized & whether or not the claim was approved.

I have 255,000 such records. Now the problem is that how do I even create the feature from this dataset. In case of differentiating between cats & dogs one can take similar sized images (lets say 100x200 pixel) which when flattened gives you a feature vector. But I cant even comprehend how to map the feature vector in my case with all these tables.

The problem is that each of the candidate has multiple claims. For example person_1 would have filed 10 claims, 6 of which approved & 4 rejected. Similarly person_2 would have filed 50, 40 of which approved and 10 rejected. How will I deal with this temporal dimension of the data ?. For comparison it would have been simpler if each person had one and only one claim and its status would have been either approved/rejected. But my data is pretty complex.

Answer 1

You could try with automated feature engineering. Featuretools is a framework to perform automated feature engineering. It excels at transforming temporal and relational datasets into feature matrices for machine learning.

https://docs.featuretools.com/

Answer 2

For each record you should create a feature vector of a fixed size.

I guess there is no problem with table candidate_general_info. You simply put the values into the vector. Or trassform them somehow - one hot encoding, create ages group etc.

From other tables you need to aggregate all rows into sigle features. For previous_employment_history table it could be has_been_unemployed, total_length_of_unemployment, average_salary, current_salary, nr_of_records_employment_history etc. The similar way for other tables.

And if I understand correctly and you want to make prediction for each claim (row of present_claim_information) each row of this table should be different datapoint (feature vector).

Answer 3

What about using some kind of recurent neural network like LSTM for the tables that can have unlimited nr of rows. This way I can embedd the variable length list of vectors (eg. previous_health_issues) to a vector of fixed size (internal state of RNN) which then can be used for the main classifier to predict claim approval.

The question is how to train the LSTMs. I am not so experienced with them but I guess it can be trained straight from main classifier with back propagation.

Or it could be trained separately.