Machine Learning Algorithm for Count or Visit data

Question

I am trying to figure out a good approach to use some machine learning on doctor appointment data. I want to first do an unsupervised clustering to look for any natural structure within the data (there are >6000 people, with >1 000 000 records total). After looking for a structure, could I also use a diagnosis as the outcome?

The first (raw) data form I can use is a record for each appointment a person has, which contains information on diagnosis, month and type of doctor. The doctor type is a flag of either Doc1 or Doc2 or neither. The simplified diagnosis code is a number between 1-20:

ID  |  Month  | Doc1_flag  |  Doc2_flag |   Dx_code
---------------------------------------------------
1       3           0             1            3
1       4           0             0            15
...     ...        ...          ....          ...
10      1           1             0            8
10      6           1             0            9

I could also roll this up to produce counts for each year, and create a variable for each diagnosis code to count per year: Dxcount1-Dxcount20. That would look like (Note - This data example assume more data than I've shown above):

ID  |  Year  | Doc1_count  |  Doc2_count |   Dxcount1 ... Dxcount20
-------------------------------------------------------------------
1       1          10            2             30             4
1       2           3            5             7              5
...     ...        ...          ....          ...    ...     ...
10      1           0            5             14             23 
10      2           12           0             9              2

If I were to roll the data up to counts per year I saw this question that I could apply to my data. Would I be able to leave the data in it's raw format and use a hierarchical clustering method? I'm fairly new to ML in it's application, so any further reading for my question would be much appreciated!

Edit: My goal is to look for natural clusters within the data, so do those with more visits to doctor 1 correspond with more diagnosis 2, etc. I could also use the diagnosis as a label for each individual appointment to predict diagnosis based on appointment.

Answer 1

You need to have a useful measure of dissimilarity first.

Once you can quantify how related two cases are, then you can try plenty of different clustering algorithms. But there is no "correct" way to measure similarity - this is a domain specific choice. For example, if you know that the data should be interpreted as histograms, then you can consider using histogram divergence measures. But if that does not make sense (and I'm not convinced that it does) then you should choose a different dissimilarity.

Answer 2

For finding natural clusters, a few approaches come to find. First, you can start simple by taking each doctor's diagnosis for a year, making a vector out of that and computing pairwise distances between all the vectors. This would give a representation of which doctors are similar to which other and which are outliers. Then you can do agglomerative/hieracical clustering to get a better sense of which doctor clusters exist.

As for predicting which doctor would give which DxCodes, either on the original data or on the data you clustered like above, you can fit any kind of supervised learning model, (logistic regression, multinomial regression, neural network, KNN regression) to try and predict what the doctors will prescribe. If you're new to ML also read up /remember to make a testing and training set for your data.