I have dataset of a company with 2014 data with 15 variables then 2018 data with same 15 variables.I want to combine both the datasets however the meaning of 1 variable has changed meaning that variable no longer has same significance in both datasets. How to deal with this variable? whats the best approach for building machine learning model in this case? Also, importance of the variable according to model summary is fairly good in both the datasets. I am building the GBM model here.

  • $\begingroup$ Can you be more explicit as to how the variable has changed over time? If the variable refers to two different things but is named the same thing, this is equivalent to TWO different variables. $\endgroup$ – M Waz Jul 8 at 20:46
  • $\begingroup$ I concur with M Waz. If the change in the variable is a change of units, then it should be easy to change the units back. If there is is a plausible way to map both variables to a third variable that has consistent meaning between the two years, then that is the way to go. Recoding variables is a very, very common task in the step-one-cleaning-the-data phase of machine learning. $\endgroup$ – Peter Leopold Jul 8 at 21:57
  • $\begingroup$ @MWaz the variable previously was used to collect if the user registered online voluntarily or they registered through a recruitment drive( basically a boolean value isRegOnline) but now even though the offline recruitment drive happens, company asks their customers to register online after collecting documents. So, its difficult to determine whether the user registered online voluntarily with self interest or through the recruitment drive. $\endgroup$ – krishna koti Jul 9 at 14:07
  • $\begingroup$ @PeterLeopold It is not the change in units and i dont think I can map both variables to a third variable. Please refer to my comment above on variable explanation. $\endgroup$ – krishna koti Jul 9 at 14:09
  • $\begingroup$ @krishnakoti The variable does not have the same meaning any more because the method for collecting the information has changed. The variable changed from did they register online or not voluntarily to did they follow the company's instructions to register online as a second step in the registration process. Even though they are coded the same, they do not refer to the same thing. $\endgroup$ – M Waz Jul 9 at 14:16

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.