Looking at the Heritage Health Prize, the data is structured as follows:
Each of the Data Sets will be comprised of tables as follows:
a. Members Table, which will include: i. MemberID (a unique member ID) ii. AgeAtFirstClaim (member's age when first claim was made in the Data Set period) iii. Sex b. Claims Table, which will include: i. MemberID ii. ProviderID (the ID of the doctor or specialist providing the service) iii. Vendor (the company that issues the bill) iv. PCP (member's primary care physician) v. Year (the year of the claim, Y1, Y2, Y3) vi. Specialty vii. PlaceSvc (place where the member was treated) viii. PayDelay (the delay between the claim and the day the claim was paid for) ix. LengthOfStay x. DSFS (days since first service that year) xi. PrimaryConditionGroup (a generalization of the primary diagnosis codes) xii. CharlsonIndex (a generalization of the diagnosis codes in the form of a categorized comorbidity score) xiii. ProcedureGroup (a generalization of the CPT code or treatment code) xiv. SupLOS (a flag that indicates if LengthOfStay is null because it has been suppressed) c. Labs Table, which will contain certain details of lab tests provided to members. d. RX Table, which will contain certain details of prescriptions filled by members. e. DaysInHospital Tables - Y2 and Y3, which will contain the number of days of hospitalization for each eligible member during Y2 and Y3 and will include: i. MemberID; ii. ClaimsTruncated (a flag for members who have had claims suppressed. If the flag is 1 for member xxx in DaysInHospital_Y2, some claims for member xxx will have been suppressed in Y1). iii. DaysInHospital (the number of days in hospital Y2 or Y3, as applicable). These two Tables are intended for use by Entrants to train and validate their algorithms. DaysInHospital Tables are based on the Claims Table with admissions in Y2 or Y3, as applicable. As a privacy measure, any member who spent more than two weeks in hospital is grouped; they are treated as though they spent 15 days in hospital. f. Target - is "DaysInHospital_Y4" but doesn't include DaysInHospital. DaysInHospital data for Y4 are to be filled in by Entrants to produce entries. Seem SampleEntry.csv as an example.
So the goal is to predict the values in table f.
Clearly the first thing to do is to do a JOIN across the tables using member IDs.
Some of these appear to be standard discrete variables (age, sex etc) but others are much more tricky to deal with. For example, the labs index and RX index point to other tables. Also, there are many categorical types. Some of the categorical types (such as PrimaryConditionGroup) have many possible values. Normally for categorical data, these can be made into features using a binary flag for each category (simply using the index will fail as there is no ordering). However when there are many thousands of these this seems like an inefficient approach. Are there any better ways to deal with the mixture of numerical and categorical data such as this?