I have access to two large medical datasets of observational records in the UK. The first - Clinical Practice Research Datalink (CPRD) - has data on 100,000's of patients - largely dates of doctor's visits, diagnoses & prescriptions. Crucially, it has little information on hospitalisations.

The second dataset is of hospitalisations - Hospital Episode Statistics (HES) - and importantly, I can identify HES patients in the CPRD. The HES data is a lot smaller - perhaps 5% of CPRD patients (I haven't the figures to hand)? - but it's still tens of thousands of patients.

The brief is to look for signals in the CPRD data that might indicate a hospitalisation - using the HES data to verify this. I'll likely do this within a cohort of patients with a specific illness (respiratory disease, for example) which should make their medical patterns more homogeneous than the general population.

The reason for the research is that CPRD is richer in medical history making it the most useful - but lacks that crucial element of hospitalisations. Still, it does have some - a patient might visit the doctor's and be told to go to ER, for example - this will be recorded in CPRD.

Ultimately, I'm not optimistic I'll find much but I need to give it a good go. My background is traditional frequentist statistics where tools such as agreement measures just seem woefullly inadequate. I have a nodding knowledge of some bayesian and machine learning techniques - textbook rather than practical - and the latter seems appropriate in this case.

I'm really looking for keywords & ideas to research (not necessarily full answers) as this will help greatly.

Also, I'll likely program this in SAS 9.1.3 (or 9.3 eventually) - if that makes any difference - though I might be able to use R (though dataset size would definitely be an issue).

I'm grateful for any help.


Resp Arthur Small: Yes, I suppose you could use CPRD records in that way, using HES records to 'confirm' hosps. I wonder though - given that CPRD is longitudinal data - at which points in CPRD you determine these probabilities (without first peeking at HES to get the date). Still, it definitely gives me something to investigate further - and is in my world of inferential stats!


Resp Peter Ellis: It is possible to use CPRD patients with HES (and the overlapping period) so yes, we can be sure that all hosps are recorded in HES for that subset of CPRD patients.

I can't see how logistic regression would work however. If a patient has a HES hosp in June and Nov of a given year, I'm not sure how regressing on CPRD events will point to 1 hosp and not the other. I think it's the time element that's confusing me. I 'anticipate' a cluster of CPRD events (and event types) around HES hosps - it's that sort of analysis I believe I need. Thanks.

  • $\begingroup$ I understand that you want to develop a predictive model that estimates the probability that a given patient will be hospitalized, as a function of characteristics described in the patient's medical history as recorded in the CPRD dataset. Yes? $\endgroup$ Dec 17, 2012 at 5:21
  • $\begingroup$ If so: I suggest looking at Probit models. The idea is to treat the probability of hospitalization as a function of an unobserved (a.k.a. latent) variable, the value of which is a linear function of the characteristics described in the CPRD record. Formally: Pr(Hospitalization = 1 | X) = f(X'b), where f() is the cumulative probability function for the normal distribution, X is a vector of CPRD characteristics, and b is a vector of parameters to be estimated, e.g., via Maximum Likelihood. $\endgroup$ Dec 17, 2012 at 5:30
  • $\begingroup$ See: en.wikipedia.org/wiki/Probit_model $\endgroup$ Dec 17, 2012 at 5:30
  • 1
    $\begingroup$ Does the HES include all hospitalisations? Or, putting it the other way, are there instances in the CPRD that were hosptialised but weren't recorded in the HES? I'm presuming that not all hospitalisations are in the HES because if they were you would have a straightforward logistic regression problem. Possibly this is the answer anyway, but it is made more complex if there are randomly missing cases where it is not known if hospitilisation happend or not. $\endgroup$ Dec 17, 2012 at 9:47
  • 1
    $\begingroup$ Just as a side note, check some references for R for using big datasets, e..g. cran.r-project.org/web/views/HighPerformanceComputing.html $\endgroup$ Dec 17, 2012 at 11:42

1 Answer 1


Machine learning idea: you could train a binary or one-class classifier of your choice to recognize what patterns in CPRD data indicate a hospitalization. You can use the HES data to label positive patterns in the CPRD data (e.g. those patterns that signify a hospitalization). I assume you could also use HES data to label negatives, if not, you'll need a one-class technique.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.