I am working on a personal rento in which based on some input data I have to predict some output data. The challenge is to predict the expenses in transactions, receipts and cards that users will have next year. For this, they provide me as input data the expenses of each client of a whole year of the three fields described above and also the sociodemographic data of each user as country of residence, age, gender, etc. I'll leave you a sample of both tables. The problem with which I find myself is that in the sociodemographic data I do not have coherent data. In the country of residence, or gendre or Age for example I have this:
Domain of Sexo -------> ['R', 'S'] Domain of Edad -------> [Min : 10, Max : 118] Domain of PAIS_RESI ------->['BH', '+V', '#B', 'GB', 'CV', ')Z','EH', 'TS', 'W)', 'RB', 'HB', 'QY', 'KC', 'DM', 'DE', 'AK', 'HD', 'KJ', 'ZH', 'EL', 'JG', 'EV', '#C', 'BT', 'EB', 'UJ', 'HJ', 'TJ', 'QJ', 'JC', 'VZ', 'GU', 'TW', 'WL', 'KB', 'DV', 'SW', 'PU', 'GL', 'GP', '+#', 'RW']
I have many more columns and the degree of inconsistency remains the same. I don't know how to clean the data in order to analyze the data. There is inconsistency because i don't know the meaning of 'BH' or '+V' in the country code.
I will provide you an image of both tables. The SocioDemo table has this format:
ID SEXO EDAD PAIS_RESI ... autonper cpaisper codine codseg 0 XXXXXKLQX R 70 BH ... XF BH FFXFN K# 1 XXXXXKQAQ R 83 BH ... XU BH YUXQU KV 2 XXXXXABUB S 73 +V ... NaN +V NaN KV 3 XXXXUUQUX S 76 BH ... XY BH XAXYX KV 4 XXXXUKXLA R 71 #B ... NaN #B NaN KV
The meaning of the columns on the sociodemo table are:
ID ---> client ID
SEXO ---> gendre
PAIS_RESI ---> Country of residence code
PAIS_NACI -----> Country of birth code
nivestud ---> educational level
idiomapr ---> preferred language
and so on...
The meaning of columns on the transcational table are:
ID -----> client ID
mes ----> month
transfers -----> transfers in a month
recibos ----> receipts
tarjeta -----> cards.
My goal is to predict and get a table like the transactional.csv for the next year. The problem is that I do not have coherent data or a set of training and testing. How do you recommend me to approach this issue? How would you treat with the inconsistency?
I'm using pandas, pyton, sklearn, openrefine.
Even with clustering i cannot agroup the set of values