2
$\begingroup$

I am trying to build a model to predict the length of time people live in a house. I have a good set of historical data (15k rows) but only about 4k of these are complete in so far as the person has moved out of the house and I have an actual length of stay. Therefore for 4k rows I know exactly how long they have stayed and for 11k rows I know 'at least' how long they stayed.

I can obviously train on just the 4k rows but I'd love to be able to take advantage of the information in the rest of my dataset. Any thoughts on how I might achieve this?

Improve this question
$\endgroup$
2
  • $\begingroup$ Ooh! I see a problem with the data. The 4k data captured the shorter residence times only. $\endgroup$
    – Carl
    Oct 5, 2016 at 0:29
  • 1
    $\begingroup$ This is essentially the problem of survival analysis. $\endgroup$
    – Danica
    Oct 5, 2016 at 0:32

2 Answers 2

Reset to default
1
$\begingroup$

I think you need to do a bit of feature engineering here- for example create a new variable as if the person still living in the house or left... or number of years staying before leaving to utilise all your data.

Improve this answer
$\endgroup$
0
$\begingroup$

What I would do rather than think too hard about the problem would be to do a Monte Carlo simulation study, and adjust the parameters of length of stay to give me 4k observed residence time and 11 k not. As things stand right now, the 4k data only sees what happened for those residence times that were short enough to be captured, so it is a biased sample. During my simulation, I would adjust the total 15k model's residence time density function so that the captured residency times it makes in the observed 4k residency window makes a statistically identical residency density function in that window to the observed one, and also a distribution of observed times in the 11k window that are consistent with that.

The advantage of doing it that way, is that my 15k model with then have residency times that are far in excess of those that can be captured by the data. Now, maybe there is a more elegant way of doing the same thing, but in general, this is a typical inverse problem, so all solutions lie in that direction.

Things that need testing include 1) What is the distribution of residence times that explains the data? 2) How well does that explain the data?

That is, before one can do anything with the data, one has to either find an empirical or actual residence time distribution that is consistent with the data in both groups. If one can determine a residence time, know distribution type model that is highly explanatory, the problem is simplified considerably.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.