# Given list symptoms and diseases - how to find similar symptoms based on data

I have a dataset of case studies like that:

Gender ; Age ; Symptom ; Condition

M ; 20 ; Fever ; Flu
F ; 29 ; Arm pain ; Trauma
F ; 40 ; Fever ; Cancer
M ; 90 ; Diarrhea ; Flue

and a model that predict the condition of each new sample (Gender + Age + Symptom). Due to the fact that each condition have many different symptoms, i would like to build an algorithm that for each sample + prediction , will give me the best symptoms that i need to check with the person to increase the certainty of the model prediction or update the prediction to other condition .

i have checked some exotic approaches such as active learning or cost sensitive learning but i'm pretty sure that there are some more basic approaches that i'm missing here.

• I think the simplest solution might be a lookup table: for each condition, rank the symptoms in order of prevalence or informativeness, and iterate between prediction with the model trained on the data and collecting additional information from the table. If you however want rigorous evaluation of probability, then you need to start with a probabilistic model for symptoms (and symptom checking) and diagnosis/disease confirmation. I'd have to think about what such a model would be. – deasmhumnha Aug 22 '18 at 20:06
• I need it in real time , so its an issue to check for each sample all the possible models .. can you elaborate more about the probabilistuc model? Thanks – Latent Aug 23 '18 at 4:36
• You should look into latent variable models. Here the latent variables would be possible but unconfirmed symptoms. You could then approximate the probability distribution over the latent variables given validated symptoms $X$ and confirmed diagnoses $Y$ as $P(Z |X,Y)$ either using EM or a Bayesian approach. I'll try to write a formal description of such a model later. I also remember reading a paper about imputing missing data in SVMs based on the similar idea, though in your case the focus is on the latent variables. – deasmhumnha Aug 23 '18 at 20:59

Following my comment above, if $X$ is our confirmed symptoms (as binary present/absent variables) and $Y$ is our diagnosis (as one-hot vector/categorical variable), the $Z$ is our untested/unconfirmed systems (also as binary variables). The simplest model would be multi-logistic regression such that $E[Y|X,Z]=\mathop{\mathrm{softmax}}([X,Z]\theta)$, but we could choose our estimator to be any function $f$ such as a neural network. Here $[X,Z]$ is sample-wise concatenation. To fit this model, we use the likelihood that $$\mathcal{L}(Y|X,Z)=\sum_i \mathrm{Cat}(Y=y_i|E[Y]=f(\theta,x_i,z_i))$$
For the frequentist solution, we use EM algorithm and any associated shortcuts or heuristics (a discussion of which is outside the scope of this post). Once we have our estimator for $\theta$ we can find the most likely $z$ given $x$ and/or $y$ and check for those symptoms with a value of 1.
A Bayesian approach would instead generate many samples of both $\theta$ and $Z$ given the above likelihood and reasonable priors. We would then take the expectations $E[Z|X=x, Y=y]$ and/or $E[Z|X=x]$ over all values of $\theta$. It might be worth it to use variational Bayes however if your training data is particularly large. You could then choose to check for symptoms with highest expectation (highest marginal probability).