I am working on a personal rento in which based on some input data I have to predict some output data. The challenge is to predict the expenses in transactions, receipts and cards that users will have next year. For this, they provide me as input data the expenses of each client of a whole year of the three fields described above and also the sociodemographic data of each user as country of residence, age, gender, etc. I'll leave you a sample of both tables. The problem with which I find myself is that in the sociodemographic data I do not have coherent data. In the country of residence, or gendre or Age for example I have this:

 Domain of Sexo -------> ['R', 'S']
 Domain of Edad -------> [Min : 10, Max : 118]
 Domain of PAIS_RESI ------->['BH', '+V', '#B', 'GB', 'CV', ')Z','EH', 'TS', 'W)', 'RB', 'HB', 'QY', 'KC', 'DM', 'DE', 'AK', 'HD', 'KJ', 'ZH', 'EL', 'JG', 'EV', '#C', 'BT', 'EB', 'UJ', 'HJ', 'TJ', 'QJ', 'JC', 'VZ', 'GU', 'TW', 'WL', 'KB', 'DV', 'SW', 'PU', 'GL', 'GP', '+#', 'RW']

I have many more columns and the degree of inconsistency remains the same. I don't know how to clean the data in order to analyze the data. There is inconsistency because i don't know the meaning of 'BH' or '+V' in the country code.

I will provide you an image of both tables. The SocioDemo table has this format:

    ID        SEXO   EDAD   PAIS_RESI  ...   autonper cpaisper codine codseg
0  XXXXXKLQX    R    70        BH  ...         XF       BH      FFXFN     K#
1  XXXXXKQAQ    R    83        BH  ...         XU       BH      YUXQU     KV
2  XXXXXABUB    S    73        +V  ...        NaN       +V        NaN     KV
3  XXXXUUQUX    S    76        BH  ...         XY       BH      XAXYX     KV
4  XXXXUKXLA    R    71        #B  ...        NaN       #B        NaN     KV

The meaning of the columns on the sociodemo table are:

ID ---> client ID

SEXO ---> gendre

PAIS_RESI ---> Country of residence code

PAIS_NACI -----> Country of birth code

nivestud ---> educational level

idiomapr ---> preferred language

and so on...

enter image description here

The meaning of columns on the transcational table are:

ID -----> client ID

mes ----> month

transfers -----> transfers in a month

recibos ----> receipts

tarjeta -----> cards.

My goal is to predict and get a table like the transactional.csv for the next year. The problem is that I do not have coherent data or a set of training and testing. How do you recommend me to approach this issue? How would you treat with the inconsistency?

I'm using pandas, pyton, sklearn, openrefine.

Even with clustering i cannot agroup the set of values


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.