1
$\begingroup$

I am trying to predict the length of time (not a time series!) a work task takes, especially technical changes. The output should be a prediction of the duration the task will take in days. As input I have the historical data with characteristics like what kind of change, priorisation, responsibles, components, categorie, how many tasks, cost relevance, ..and some more.. and of course the duration the change lasted.

So after I prepared the data, I am thinking what would be the best way to solve this with machine learning. I thought about linear regression or should I first do a clustering and then regression? My thoughts relating to the clustering were that I can first characterize similar tasks and then do the regression. Or would a multiple regression be the better way? Do you have any suggestions based on your experiences?

$\endgroup$
1
  • 2
    $\begingroup$ What does it have to do with clustering? Seems like a clear regression problem. Also, is this a python specific question? I.e. are you asking for a method recommendation or for code? $\endgroup$ – Jan Kukacka Jun 29 '18 at 7:02
1
$\begingroup$

This is a classic survival analysis situation. There are a large number of methods for this. It may help you to read through some of the threads categorized under our tag. In particular, you might want to see: Survival Analysis tools in Python. Clustering could certainly be relevant, in the sense that it always could be, but there is no reason to expect it is necessary or especially useful here.

$\endgroup$
1
  • $\begingroup$ Thank you so much, this is great! I read about survival analysis after reading your post and it is exactly what I am looking for. I will try it out now $\endgroup$ – lambkin Jun 29 '18 at 20:52
0
$\begingroup$

Doing first clustering and then regression makes only sense, if you want to have information about the groups identified with the cluster. If you want information for each category you can just include each one as dummy. you can try clustering and see if your identified clusters make sense from a topic point of view (this is highly dependent on your data and a quite sensitive approach). If you have a reasonable amount of categories using the dummy approach within regression might be more soundproof.

I don't know how many variables you have, but if there are a lot, you might consider adding a lasso-penalty term. In that way, some of your coefficients that are not significant will be shrunken to zero (and not included in your linear regression) and that way you will avoid overfitting your data.

Another possibility which often works quite well might be to use random forest regression. There, the number of input variables doesn't matter and you often get good prediction results as well.

Just remember, that for lasso and for random forest you have to tune your parameter accordingly!

$\endgroup$
3
  • $\begingroup$ Hi, thanks for your suggestions! I have a lot to think about and google now :-) I have 13 variables at the moment and the target. What number is recommendable for lasso-penality? $\endgroup$ – lambkin Jun 28 '18 at 21:23
  • $\begingroup$ What do you mean what number is recommended? You mean to tune the lambda? How many observations do you have? Lambda is between 0 and 1, depending on how big your data set is you should try as many values as possible ( I would start off with 10 possible lambdas and then when you see that the best lambda is between lets say 0.1 and 0.2 you can go deeper and try 0.01, 0.03, ... $\endgroup$ – LN_P Jun 28 '18 at 21:31
  • $\begingroup$ dont forget to use cross validation (or at least a train and test set) $\endgroup$ – LN_P Jun 28 '18 at 21:32

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.