I have a data-set/data-frame with columns Description, Department Name, Priority, Doctor name & Location.Description data comes as free text from from the UI.

Based on historical description's in the data-set, my data scientist (DS) says he can predict the Department Name value for a new incoming Description. However he says he cannot predict Priority, Doctor Name and Location!

Based on the historical data analysis/ML/mining, I understand the DS has come to that conclusion.

As someone aspiring to learn ML, what analysis/algorithms does one run on a data-set to determine which features can be predictable, which cannot?

  • $\begingroup$ Please give further data: as it is stated to predict the doctor name one should know his parents... If you can provide also a snapshot of some rows of this dataset :) $\endgroup$ – Tommaso Guerrini Mar 26 '17 at 8:09
  • $\begingroup$ I was generically giving this schema/data-frame as an example...I am interested in knowing what makes a column in general predictable while some columns cannot be predicted. i.e. let's say based on column A (text column) we can predict column B (text column) result; but based on column A the data scientist says they cannot predict column C (also a text column). What analysis does one to do say that column C contents cannot be predicted based on column A....does the clarify? thanks. $\endgroup$ – AreForRavi Mar 26 '17 at 8:21

OK. Found out that specifically for this schema priority, doctor name and location could be dependent on 10 other things and not just on description column. In general though just because column B can be predicted based on column A values, does not mean column C, column D etc. can be predicted even though all are of string/nvarchar type. Reason being, column B is directly dependent on column A, while C, D, etc. columns could be influenced by other attributes not even present in the data-set/data-frame. Also, if prediction algorithm results shows that if description data is "some medical condition" and priority was found to be "P2" from historical data in 90% of the data set then the column priority can be predictable.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.