Predict duration of tasks with clustering and regression combined in Python

Question

I am trying to predict the length of time (not a time series!) a work task takes, especially technical changes. The output should be a prediction of the duration the task will take in days. As input I have the historical data with characteristics like what kind of change, priorisation, responsibles, components, categorie, how many tasks, cost relevance, ..and some more.. and of course the duration the change lasted.

So after I prepared the data, I am thinking what would be the best way to solve this with machine learning. I thought about linear regression or should I first do a clustering and then regression? My thoughts relating to the clustering were that I can first characterize similar tasks and then do the regression. Or would a multiple regression be the better way? Do you have any suggestions based on your experiences?

Answer 1

This is a classic survival analysis situation. There are a large number of methods for this. It may help you to read through some of the threads categorized under our survival tag. In particular, you might want to see: Survival Analysis tools in Python. Clustering could certainly be relevant, in the sense that it always could be, but there is no reason to expect it is necessary or especially useful here.

Answer 2

Doing first clustering and then regression makes only sense, if you want to have information about the groups identified with the cluster. If you want information for each category you can just include each one as dummy. you can try clustering and see if your identified clusters make sense from a topic point of view (this is highly dependent on your data and a quite sensitive approach). If you have a reasonable amount of categories using the dummy approach within regression might be more soundproof.

I don't know how many variables you have, but if there are a lot, you might consider adding a lasso-penalty term. In that way, some of your coefficients that are not significant will be shrunken to zero (and not included in your linear regression) and that way you will avoid overfitting your data.

Another possibility which often works quite well might be to use random forest regression. There, the number of input variables doesn't matter and you often get good prediction results as well.

Just remember, that for lasso and for random forest you have to tune your parameter accordingly!