I am trying to figure out a good approach to use some machine learning on doctor appointment data. I want to first do an unsupervised clustering to look for any natural structure within the data (there are >6000 people, with >1 000 000 records total). After looking for a structure, could I also use a diagnosis as the outcome?

The first (raw) data form I can use is a record for each appointment a person has, which contains information on diagnosis, month and type of doctor. The doctor type is a flag of either Doc1 or Doc2 or neither. The simplified diagnosis code is a number between 1-20:

ID  |  Month  | Doc1_flag  |  Doc2_flag |   Dx_code
1       3           0             1            3
1       4           0             0            15
...     ...        ...          ....          ...
10      1           1             0            8
10      6           1             0            9

I could also roll this up to produce counts for each year, and create a variable for each diagnosis code to count per year: Dxcount1-Dxcount20. That would look like (Note - This data example assume more data than I've shown above):

ID  |  Year  | Doc1_count  |  Doc2_count |   Dxcount1 ... Dxcount20
1       1          10            2             30             4
1       2           3            5             7              5
...     ...        ...          ....          ...    ...     ...
10      1           0            5             14             23 
10      2           12           0             9              2

If I were to roll the data up to counts per year I saw this question that I could apply to my data. Would I be able to leave the data in it's raw format and use a hierarchical clustering method? I'm fairly new to ML in it's application, so any further reading for my question would be much appreciated!

Edit: My goal is to look for natural clusters within the data, so do those with more visits to doctor 1 correspond with more diagnosis 2, etc. I could also use the diagnosis as a label for each individual appointment to predict diagnosis based on appointment.

  • $\begingroup$ Have you label-encoded your diagnosis codes manually? Is there a ranking / ordinality between the different diagnoses, or are they nominal (like in diag1 = flu, diag2= common cold, etc.)? $\endgroup$ – Zhubarb May 7 '19 at 13:20
  • $\begingroup$ Oddly enough, it could be treated as either. Part of what I'm trying to answer is "Is diag1 more severe than diag2, etc", and it may be helpful to test both ways. Are there two different approaches depending on ordinality? The ranks are somewhat of a "best guess" based on medical knowledge and need to be compared. $\endgroup$ – Yolo_chicken May 7 '19 at 13:31
  • $\begingroup$ If it was nominal, you can run a classifier with the dependent variable (label) having a cardinality of 20. When it is ordinal, you will need to bear in mind that classification will lose that nuance. The nature of the diagnosis code is important not just in classification, but also in clustering, etc. It is the feature representation step which is a prerequisite before feeding your data into any algorithm. $\endgroup$ – Zhubarb May 7 '19 at 13:46
  • $\begingroup$ Would you mind saying a bit more about what your goal is? Machine learning can be used in many many different ways, but without a clearer goal it's a bit hard to advise. Are you looking for correlation patterns ie (if patient A went to doctor B, is he more likely to do to doctor C or D)? Or is it classification (trying to see how the outcomes of patients look like after going to doctor A vs doctor B)? $\endgroup$ – Anonymous Emu May 7 '19 at 15:05
  • $\begingroup$ I mostly want to look for any natural clusters in the data. So if a person goes to doctor A often, are they more likely to have more counts of dxcode 1. I would also want to do similar to what you mentioned, where if a person sees doctor B, are they likely to be a specific dxcode. Does this make my goal clear? $\endgroup$ – Yolo_chicken May 7 '19 at 15:12

You need to have a useful measure of dissimilarity first.

Once you can quantify how related two cases are, then you can try plenty of different clustering algorithms. But there is no "correct" way to measure similarity - this is a domain specific choice. For example, if you know that the data should be interpreted as histograms, then you can consider using histogram divergence measures. But if that does not make sense (and I'm not convinced that it does) then you should choose a different dissimilarity.

  • $\begingroup$ Where I'm new and trying to understand the steps for a ML learning algorithm: do most ML algorithms require data to be placed in a dissimilarity/similarity matrix when using nominal/ordinal/categorical data? I would imagine continuous data does not require a matrix form but I could be wrong. $\endgroup$ – Yolo_chicken May 9 '19 at 11:42
  • $\begingroup$ HAC is defined on a distance matrix. DBSCAN does not, but if you have some unusual, custom distance matrix it is the easiest way to use a precomputed distance matrix. But by any means, you do need a notion of similarity. $\endgroup$ – Has QUIT--Anony-Mousse May 9 '19 at 14:43

For finding natural clusters, a few approaches come to find. First, you can start simple by taking each doctor's diagnosis for a year, making a vector out of that and computing pairwise distances between all the vectors. This would give a representation of which doctors are similar to which other and which are outliers. Then you can do agglomerative/hieracical clustering to get a better sense of which doctor clusters exist.

As for predicting which doctor would give which DxCodes, either on the original data or on the data you clustered like above, you can fit any kind of supervised learning model, (logistic regression, multinomial regression, neural network, KNN regression) to try and predict what the doctors will prescribe. If you're new to ML also read up /remember to make a testing and training set for your data.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.