# Machine Learning Algorithm for Count or Visit data

I am trying to figure out a good approach to use some machine learning on doctor appointment data. I want to first do an unsupervised clustering to look for any natural structure within the data (there are >6000 people, with >1 000 000 records total). After looking for a structure, could I also use a diagnosis as the outcome?

The first (raw) data form I can use is a record for each appointment a person has, which contains information on diagnosis, month and type of doctor. The doctor type is a flag of either Doc1 or Doc2 or neither. The simplified diagnosis code is a number between 1-20:

ID  |  Month  | Doc1_flag  |  Doc2_flag |   Dx_code
---------------------------------------------------
1       3           0             1            3
1       4           0             0            15
...     ...        ...          ....          ...
10      1           1             0            8
10      6           1             0            9


I could also roll this up to produce counts for each year, and create a variable for each diagnosis code to count per year: Dxcount1-Dxcount20. That would look like (Note - This data example assume more data than I've shown above):

ID  |  Year  | Doc1_count  |  Doc2_count |   Dxcount1 ... Dxcount20
-------------------------------------------------------------------
1       1          10            2             30             4
1       2           3            5             7              5
...     ...        ...          ....          ...    ...     ...
10      1           0            5             14             23
10      2           12           0             9              2


If I were to roll the data up to counts per year I saw this question that I could apply to my data. Would I be able to leave the data in it's raw format and use a hierarchical clustering method? I'm fairly new to ML in it's application, so any further reading for my question would be much appreciated!

Edit: My goal is to look for natural clusters within the data, so do those with more visits to doctor 1 correspond with more diagnosis 2, etc. I could also use the diagnosis as a label for each individual appointment to predict diagnosis based on appointment.

• Have you label-encoded your diagnosis codes manually? Is there a ranking / ordinality between the different diagnoses, or are they nominal (like in diag1 = flu, diag2= common cold, etc.)? – Zhubarb May 7 '19 at 13:20
• Oddly enough, it could be treated as either. Part of what I'm trying to answer is "Is diag1 more severe than diag2, etc", and it may be helpful to test both ways. Are there two different approaches depending on ordinality? The ranks are somewhat of a "best guess" based on medical knowledge and need to be compared. – Yolo_chicken May 7 '19 at 13:31
• If it was nominal, you can run a classifier with the dependent variable (label) having a cardinality of 20. When it is ordinal, you will need to bear in mind that classification will lose that nuance. The nature of the diagnosis code is important not just in classification, but also in clustering, etc. It is the feature representation step which is a prerequisite before feeding your data into any algorithm. – Zhubarb May 7 '19 at 13:46
• Would you mind saying a bit more about what your goal is? Machine learning can be used in many many different ways, but without a clearer goal it's a bit hard to advise. Are you looking for correlation patterns ie (if patient A went to doctor B, is he more likely to do to doctor C or D)? Or is it classification (trying to see how the outcomes of patients look like after going to doctor A vs doctor B)? – Anonymous Emu May 7 '19 at 15:05
• I mostly want to look for any natural clusters in the data. So if a person goes to doctor A often, are they more likely to have more counts of dxcode 1. I would also want to do similar to what you mentioned, where if a person sees doctor B, are they likely to be a specific dxcode. Does this make my goal clear? – Yolo_chicken May 7 '19 at 15:12