Oversimplifying a bit, I have about a million records that record the entry time and exit time of people in a system spanning about ten years. Every record has an entry time, but not every record has an exit time. The mean time in the system is ~1 year.
The missing exit times happen for two reasons:
- The person has not left the system at the time the data was captured.
- The person's exit time was not recorded. This happens to say 50% of the records
The questions of interest are:
- Are people spending less time in the system, and how much less time.
- Are more exit times being recorded, and how many.
We can model this by saying that the probability that an exit gets recorded varies linearly with time, and that the time in the system has a Weibull whose parameters vary linearly with time. We can then make a maximum likelihood estimate of the various parameters and eyeball the results and deem them plausible. We chose the Weibull distribution because it seems to be used in measuring lifetimes and is fun to say as opposed to fitting the data better than say a gamma distribution.
Where should I look to get a clue as to how to do this correctly? We are somewhat mathematically savvy, but not extremely statistically savvy.